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Abstract 

The  study  explores  issues  universities  consider  when  deciding  what  types  of  assessments 
will  be  used  for  distance  education  students.  Many  universities  conduct  distance  education 
courses  electronically  but  frequently  do  not  execute  assessments  electronically.  Universities 
across  the  country,  diversified  by  academic  rankings,  tuition,  and  student  population  were 
interviewed  for  this  study. 

Results  generally  revealed  security  concerns  as  the  most  prevalent  reason  for  universities 
avoiding  electronic  assessments.  However,  some  universities  created  evaluation  methods  to 
mitigate  the  security  risks  and  performed  electronic  assessments  such  as  projects  or  exams 
designed  to  reduce  the  possibility  of  cheating.  Examples  of  these  types  of  assessments  include 
requiring  students  to  interpret  material  discussed  during  the  course,  using  questions  that  require 
more  than  looking  up  the  answers  in  course  material,  or  asking  personal  questions  that  only  the 
student  should  know.  These  are  not  foolproof,  but  do  lessen  the  risk  of  cheating. 

A  majority  of  the  universities’  representatives  indicated  faculty  tends  to  resist  changes, 
particularly  technological  changes.  Several  of  the  schools  have  faculties  that,  for  the  most  part, 
are  not  connected  to  a  network  or  utilize  electronic  mail.  Consequently,  technological 
limitations  are  restricting  the  potential  capabilities  of  the  faculty.  However,  the  universities’ 
representatives  implied  that  if  the  faculty  were  given  the  resources  they  would  still  tend  to  resist. 

The  author’s  recommendation  for  universities  is  to  reevaluate  their  perceptions  of  the 
students  and  the  potential  capabilities  that  new  technologies  can  provide  to  the  curriculum. 


STUDENT  ASSESSMENT  SELECTION  BEHAVIOR  ANALYSIS  OF 
INSTITUTIONS  CONDUCTING  DISTANCE  EDUCATION 


I.  Introduction 


General  Issue 

Across  the  United  States,  thousands  of  students  enroll  in  courses  via  distance  education. 
All  students  have  their  individual  reasons  for  selecting  these  courses,  but  convenience  is  the  most 
common  reason.  If  convenience  is  the  primary  reason  for  choosing  a  course  via  distance 
education,  why  is  the  entire  course,  assessment  included,  not  completed  in  the  same  fashion? 

The  research  demonstrates  that  typically,  a  course  conducted  electronically  will  have  a  paper- 
based,  proctored  assessment.  Communication  with  the  instructor  is  accomplished  through 
electronic  mail,  chat  rooms,  or  bulletin  boards.  For  example,  a  student  could  receive  course 
materials  and  instruction  in  electronic  format  such  as  an  Internet-based  course.  However,  when 
the  assessment  is  required,  a  student  must  find  an  acceptable  proctor  to  witness  the  execution  of 
the  graded,  paper-based  examination.  Is  this  necessary? 

There  are  technologies  and  strategies  that  can  reduce  the  opportunity  and  therefore  the 
likelihood  of  cheating.  Have  institutions  considered  these  strategies  and  technologies?  Or,  is  it 
possible  universities  have  evaluated  the  technologies  and  strategies  and  are  still  unwilling  to  risk 
the  potential  embarrassment  of  awarding  credit  for  a  course  a  student  did  not  complete  honestly? 
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The  literature  does  not  reveal  the  answer  to  this  issue.  However,  to  complete  the 
research,  a  preceding  question  was  asked.  What  issues  do  academic  institutions  consider  when 
deciding  what  types  of  course  assessments  are  acceptable  in  the  realm  of  distance  education? 

This  question  should  directly  reveal  why  a  diverse  group  of  universities  provides  various  types  of 
distance  education  courses  and  yet,  as  a  whole,  does  not  conduct  assessments  the  same  way 
(Burgess,  1994;  Spille  and  others,  1997).  Are  institutions  convinced  that  students  will  simply 
cheat  or  is  there  another  reason  for  this  apparent  disconnect?  The  preponderance  of  academic 
institutions  in  the  United  States  use  paper-based,  proctored  assessments.  These  assessments  are 
typically  created,  administered,  and  evaluated  by  the  faculty  or  representative.  The  primary  goal 
of  an  assessment  is  to  determine  whether  a  student  achieved  the  desired  level  of  understanding 
and  has  the  ability  to  apply  the  information  learned  (Brigance  and  Hargis,  1993;  Jacobi  and 
others,  1987;  Kean,  1994). 

The  idea  of  distance  education  is  not  new.  Over  the  years,  students  have  used  videotapes, 
television,  or  similar  methods  of  one-way  communication  to  receive  the  course  instruction. 
Recently,  technology  has  provided  the  ability  and  the  impetus  to  use  video-teleconferencing, 
two-way  audio  and  video  interactive  lectures,  and  Internet  courses  as  viable  methods  of 
delivering  course  instruction  to  the  student. 

Background 

Distance  education  has  been  around  for  generations  with  its  roots  in  correspondence 
programs.  The  United  States  Naval  War  College  established  the  College  of  Continuing 
Education  in  1914  to  provide  naval  officers  stationed  around  the  world  with  a  method  of 
completing  their  educational  requirements  (Stark,  1997).  Many  other  educational  institutions 
have  created  these  types  of  programs  as  well.  In  the  years  following,  videotapes  and  television 
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courses  were  the  method  of  choice  for  distributing  course  instruction  to  those  at  distant  locations. 
Now,  videotapes  and  television  are  supplementing  the  Internet,  two-way  video,  and  audio 
communication  links  (Blumenstyk,  1997;  Turtoff,  1995;  Sedlak  and  Cartwright,  1997).  To 
provide  these  new  delivery  methods,  the  public  and  private  sectors  are  spending  hundreds  of 
millions  of  dollars  to  enhance  their  communications  capabilities  and  computers  available  for 
distance  education  (Mojkowski,  1990;  White,  1990). 

Not  all  institutions,  however,  have  embraced  this  new  education  practice.  One  study 
estimates  that  55%  of  the  2,515  U.S.  four-year  colleges  provide  courses  using  distance  education 
(Gubemick  and  Ebeling,  1997).  Yale  and  Harvard  have  taken  a  “hands-off  approach”  to  this 
new,  nontraditional  teaching  method.  Yale’s  representative  argues  that  distance  education  does 
not  meet  its  requirements  for  quality.  The  school  is  not  sure  the  student  will  receive  the  same 
education  provided  to  students  attending  classes  in  residence  (Blumenstyk,  1997).  Harvard’s 
view  is  that  the  educational  experience  as  a  whole  matters  as  much  as  the  actual  classroom 
instruction.  The  dorm  life,  camaraderie,  and  personal  associations  built  during  the  college  years 
are  worth  more  than  the  convenience  of  pursuing  an  education  through  distance  learning 
(Blumenstyk,  1 997).  Harvard’s  view  of  dorm  life  and  other  aspects  of  the  college  experience 
may  be  dated.  In  1972, 28%  of  all  college  students  were  over  25.  However,  by  1994, 41%  of 
the  students  were  over  25.  This  is  indicative  of  the  target  audience  for  distance  learning 
(Gubemick  and  Ebeling,  1997).  Consequently,  the  pedagogical  approach  may  need  to  be 
different  between  the  students  in  distance  education  courses  and  the  students  in  on-campus 
courses. 

However,  some  educators  are  concerned  that  distance  education  will,  in  the  end,  result  in 
a  reduced  presence  of  teachers  (Monaghan,  1996).  One  outcome  of  this  direction  is  the  idea  of 
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stored  courses.  Stored  courses  are  those  created  and  stored  in  a  virtual  library  for  later  use  on  the 
Internet  or  transmitted  via  microwave  or  television  (Monaghan,  1996).  The  virtual  library 
would  be  like  a  repository  of  modular  portions  of  courses  that  could  be  tailored  or  grouped  to 
form  a  specific  course.  These  same  educators  wonder  whether  this  will  lead  to  students 
completing  courses  without  contact  with  an  instructor.  This  concept  is  termed  “college  in  a  box” 
(Monaghan,  1996:A23).  There  are  also  those  that  see  the  more  interactive  approach  of  the 
electronic  classroom  as  an  advantage  over  the  more  “passive”  lecture  style  of  the  traditional 
classroom  (Monaghan,  1996). 

Today’s  focus  on  distance  education  concentrates  on  the  use  of  computer  technology.  It 
stresses  interactions  between  students  and  between  students  and  the  instructor  through  the  use  of 
computers  (Sedlak  and  Cartwright,  1 997).  One  reality  for  educational  institutions  is  the  resource 
cost  of  maintaining  sufficient  communication  links  and  equipment  to  operate  the  courses 
transmitted  on  those  links.  The  cost  of  conducting  courses  and  transmitting  them  through 
satellites  and  other  media  is  quite  high.  The  plan  from  the  perspective  of  the  university  should 
include  an  analysis  of  where  it  can  get  the  most  return  from  the  investment  in  course  and 
assessment  development.  If  a  class  is  seldom  used  or  has  few  enrollees,  the  school  may  realize  a 
net  loss  for  the  particular  course.  Consequently,  schools  should  be  selective  in  the  courses  they 
offer  via  distance  education  (Sedlak  and  Cartwright,  1997).  At  Duke  University’s  Fuqua  School 
of  Business,  students  located  as  far  away  as  Switzerland  and  Hong  Kong  are  paying  $82,500  for 
a  four-year  degree.  The  same  on-campus  degree  costs  $50,000.  The  differences  in  degree  costs 
reflect  the  additional  expenses  of  hotel  accommodations  and  classroom  fees  at  sites  around  the 
world  (Gubemick  and  Ebeling,  1997). 
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A  study  identified  several  variables  influencing  the  successful  distance  education  student 
and  the  traditional  student.  Theses  variables  are  motivation,  tenacity,  desirability,  computer 
knowledge,  availability  of  Internet  capability,  and  self-discipline  (Hiltz,  1995).  During  a  three- 
year  study,  several  interesting  results  were  observed  (Hiltz  and  Wellman,  1997).  Students 
mastered  the  course  work  equally  as  well  or  better  in  the  electronic  classroom  than  in  the 
traditional  classroom.  Students  rated  access  to  faculty  and  overall  quality  of  the  educational 
experience  much  higher  in  the  electronic  classroom.  Students  tended  to  classify  the  electronic 
classroom  experience  as  group  learning  versus  individual  learning.  As  this  tendency  increases, 
the  student  is  more  likely  to  view  the  experience  as  superior  to  the  traditional  classroom  (Hiltz 
and  Wellman,  1997;  Hiltz,  1995). 

A  study  conducted  at  the  New  Jersey  Institute  of  Technology  used  pre-  and  post-course 
questionnaires,  direct  observation,  interviews  with  faculty  and  students,  and  grades  from  course 
work  to  make  comparisons  of  distance  education  and  traditional  students  by  matching  results 
from  the  same  course  taught  as  a  traditional  course  and  taught  via  distance  education.  Most 
students  (71%)  believed  they  had  better  access  to  faculty  in  the  distance  education  courses  versus 
the  traditional  classrooms.  In  addition,  73%  of  the  students  indicated  the  distance  education 
course  experience  was  more  convenient  than  the  traditional  classroom  (Hiltz  and  Wellman, 

1997;  Gillespie,  1997).  At  the  University  of  Phoenix,  all  graduating  students  were  given 
standardized  tests.  Students  educated  via  distance  education  scored  from  5%  to  10%  higher  than 
those  students  educated  in  the  traditional  classroom  (Gubemick  and  Ebeling,  1997). 

On  the  down  side,  only  33%  of  the  distance  education  students  said  they  developed  new 
friendships  during  the  distance  education  courses  while  students  in  traditional  classes  were  much 
more  likely  to  develop  friendships.  Not  surprisingly,  52%  of  the  distance  education  students 
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reported  the  tendency  to  get  busy  with  other  things  and  fall  behind  in  the  course  work.  This  last 
aspect  is  more  prevalent  in  distance  education  than  in  the  traditional  classroom  (Hiltz  and 
Wellman,  1997). 

Clearly,  academic  institutions  are  designing  courses  and  entire  curricula  for  students  to 
complete  outside  of  the  traditional  classroom.  However,  the  students  cannot  normally  complete 
the  assessment  the  same  way.  A  student  will  enroll  and  accomplish  the  course  via  distance 
education  but  when  an  assessment  is  required,  the  student  must  find  someone  to  proctor  the 
assessment.  Many  institutions,  such  as  Pennsylvania  State  University,  have  extraordinary  rules 
governing  the  occupation  and  function  of  a  proctor  (PSU,  1998). 

Problem  Statement 

From  a  review  of  the  literature,  there  are  many  accredited  and  respected  educational 
universities  offering  distance  education  courses.  Given  this  trend,  the  literature  also  reveals  a 
significant  majority  of  the  same  group  of  educational  institutions  avoid  using  nontraditional 
student  assessment  technologies,  such  as  exams  via  the  Internet.  Institutions  could  be  missing  a 
viable  and  cost  effective  method  of  conducting  assessments.  Consequently,  what  specific  issues 
are  influencing  an  institution’s  decision  to  limit  the  type  of  student  assessments  and  delivery 
methods? 

Research  Objective 

The  completed  research  will  reveal  the  issues  universities’  representatives  feel  are 
important  when  deciding  on  the  types  of  assessments  and  delivery  methods.  Additionally,  an 
assessment  selection  model  is  developed  and  tested  using  the  literature  and  the  interviewing 
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process.  The  completed  model  is  then  used  to  provide  insight  into  universities’  assessment 
selection  process. 

Research  Questions 

Does  the  Assessment  Selection  Model  (ASM)  provide  insights  into  the  specific  issues 
influencing  an  institution's  decision  on  the  type  of  student  assessments  and  the  acceptable 
delivery  method? 

Are  there  additional  issues  that  institutions  are  considering  that  the  ASM  did  not  address? 
What  is  the  relative  importance  among  the  issues  that  institutions  feel  are  significant? 

Management  Implications 

Academic  institutions  should  base  the  assessment  selection  decision  on  informed  criteria. 
Fundamentally,  a  university’s  decision  should  be  based  on  up-to-date  information  concerning 
technology,  acceptable  policies,  student  requirements,  academic  quality,  and  academic  integrity. 
This  research  could  serve  as  the  impetus  for  institutions  to  re-evaluate  their  current  assessments 
and  the  assessment  selection  process. 

Summary 

The  following  chapters  provide  additional  information  concerning  this  research  effort. 
Chapter  II  provides  the  literature  foundation  of  the  Assessment  Selection  Model.  Chapter  III 
provides  information  on  the  research  methodology  used  in  the  study.  Chapter  IV  describes  the 
data  resulting  from  conducting  the  research  as  described  in  the  methodology.  Chapter  V 
interprets  the  findings  derived  from  the  research  and  summarizes  the  thesis  as  a  whole. 
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II.  Literature  Review 


Introduction 

In  this  chapter,  the  literature  review  identifies  essential  constructs  to  create  a  model  that 
can  then  be  used  to  reveal  insights  into  an  institution's  assessment  selection  methodology.  The 
literature  concerning  assessments  covers  a  wide  range  of  issues.  However,  the  literature  did  not 
specifically  address  the  issues  institutions  use  to  establish  acceptable  assessments  and  the 
associated  delivery  methods.  Consequently,  a  model  of  the  assessment  selection  process  in 
universities  does  not  currently  exist.  To  alleviate  this,  the  current  study  creates  a  model  based  on 
similar  issues  as  discussed  in  the  literature. 

The  literature  focuses  a  great  deal  on  methods  to  develop,  validate,  and  use  assessments. 
The  material  also  discusses  the  positive  and  negative  aspects  of  the  various  types  of  assessment 
questions  including  ways  to  improve  assessments.  Additionally,  the  literature  review  reveals 
research  on  the  use  of  computers  in  the  classroom.  The  research  discusses  computer  usage  as  it 
benefits  instruction  or  usage  as  a  training  tool  as  opposed  to  benefiting  student  assessments 
(Spencer,  1996;  Crumb,  1990).  The  research  also  reflects  faculty  computer  use  as  essentially 
automating  current  instruction  methods  and  not  affecting  pedagogy. 

Discussion  of  Literature 

The  following  definition  indicates  the  purpose  of  a  well-developed  assessment  in  a 
course.  The  definition  describes  the  role  of  the  assessment  as  a  tool  to  provide  benefits  to  the 
student  and  instructor  in  terms  of  learning  and  development  and  not  simply  for  grades. 
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Assessment  is  the  process  of  defining,  selecting,  designing,  collecting, 
analyzing,  interpreting,  and  using  information  to  increase  students' 
learning  and  development.  It  includes  discussions  about  what  should  be 
assessed  and  how  information  will  be  used,  not  just  the  hands-on  testing 
of  students.  (Ratcliff,  1997:22) 

The  literature  revealed  many  issues  important  to  areas  related  to  pedagogy,  computers, 
cheating,  and  assessments.  The  particular  issues  discussed  in  this  thesis  were  chosen  for  their 
potential  usefulness  in  the  development  of  an  assessment  selection  model.  Only  through  the 
forthcoming  validity  process  can  the  usefulness  of  the  issues  be  known.  The  issues  extracted 
from  the  literature  were  grouped  into  five  broad  issues.  The  issues  are:  technology, 
administrative,  inertia,  security,  and  cost. 

Technology  issues  are  concerned  with  the  technological  capabilities  available  to  the 
institution  to  deliver  the  assessment  to  the  student.  Administrative  issues  are  those  that  address 
the  faculty’s  time  and  efforts  to  create,  maintain,  deliver,  and  score  an  assessment.  These  issues 
also  include  all  of  the  features  required  of  an  academically  rigorous  assessment.  Inertia  is  the 
tendency  of  an  institution  to  resist  changes  and  maintain  the  status  quo.  It  also  encompass  the 
institution's  concern  with  enhancing  or  maintaining  its  academic  reputation.  Security  issues  are 
concerned  with  the  institution  providing  a  secure  environment  for  the  creation,  storage,  delivery, 
scoring,  and  execution  of  an  assessment.  The  last  issue,  cost,  addresses  both  the  assessment 
costs  and  the  costs  required  to  provide  the  assessment  to  the  students. 

The  issues  listed  in  Table  1  are  the  terms  uncovered  during  the  research  of  the  literature. 
The  mapping  of  the  issues  to  the  constructs  is  as  follows: 
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Table  1.  Assessment  Selection  Model  Issues 


Constructs 

Issues 

Source 

Technology 

Issues 

Future  use  of  a  technology 

Garcia  and  Ratcliff,  1997 
Bicanich  and  others,  1997 
Burgess,  1994 

Spille  and  others,  1997 

Sufficient  technical  knowledge 

Ellington  and  others,  1993 
Farmer,  1997 

Gillespie,  1997 

Komives  and  Peterson,  1 997 
McDaniels,  1997 

Engstrom,  1997 

Bicanich  and  others,  1997 

Internet  access  AND 

Physical  transmission  of  assessment 

CUSE,  1997 

Hansen,  1994 

Kean,  1994 

IDS,  1994 

Bicanich  and  others,  1997 
Garcia  and  Ratcliff,  1997 

Web-capable  software 

Garcia  and  Ratcliff,  1997 
Gubemick  and  Ebeling,  1997 

Administrative 

Issues 

Faculty  perception  of  computer-based  testing 

Stager  and  Mueller,  1991 
Bicanich  and  others,  1997 

Assessment  Development  AND 

Ease  of  assessment  maintenance  AND 

Ease  of  assessment  scoring 

Stager  and  Mueller,  1991 
Brigance  and  Hargis,  1993 
CUSE,  1997 

Wubbels  and  Girgus,  1997 
Hansen,  1994 

Course  material  coverage 

Brigance  and  Hargis,  1993 
CUSE,  1997 

Stager  and  Mueller,  1991 

Assessment  Difficulty 

CUSE,  1997 

Cizek,  1997 

Delivery  Methods 

Bicanich  and  others,  1997 

Validity  AND 

Reliability 

Bicanich  and  others,  1997 
Brigance  and  Hargis,  1993 
Ellington,  1993 

Jacobs  and  Chase,  1992 

Inertia 

Pedagogy  changes 

Stager  and  Mueller,  1991 
GMAC,  1998 

GRE,  1998 

Hansen,  1994 

Kean,  1994 

Farmer,  1997 
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Pedagogy  changes  (con’t) 


Reputation 


Strategic  Vision 


Academic  rigor 


Institution  politics 


Mojkowski,  1990 
White,  1990 
Engstrom,  1997 
Wubbels  and  Girgus,  1997 
CUSE,  1997 
Phye,  1997 


Jacobi  and  others,  1987 


Jacobi  and  others,  1987 
Doherty  and  others,  1997 
Kean,  1994 


Hansen,  1994 
Doherty  and  others,  1997 
Thomson  and  Morse,  1997 


Spencer,  1991 
Hansen,  1994 
CUSE,  1997 
Kean,  1994 


Jacobi  and  others,  1987 
Komives  and  Petersen,  1997 
Gubemick  and  Ebeling,  1997 


Security  Issues 

Security  software  and  hardware  configurations 

Bicanich  and  others,  1997 

Password/restrict  access 

Bicanich  and  others,  1 997 

Assessment  security 

Bicanich  and  others,  1997 

Physical  environment 

Gwinn  and  Beal,  1988 
Bicanich  and  others,  1997 

Cheating 

Aiken,  1991 

Harpp  and  others,  1996 
Barnett  and  Dalton,  1981 
Bellezza  and  Bellezza,  1989 
McCabe  and  Trevino,  1996 
CUSE,  1997 

Cost  Issues 

Delivery  costs 

Hansen,  1994 

Farmer,  1997 

Hardware/software  costs 

Bicanich  and  others,  1997 

Assessment  development  and  maintenance  costs 

Bicanich  and  others,  1997 

Costs  to  Implement  Strategic  Plan 

Ferren,  1997 

Technology  Issues 

Technology  issues  are  concerned  with  the  technological  capabilities  available  to  the 
institution  to  deliver  the  assessment  to  the  student.  Additionally,  an  institution  must  determine 
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what  types  of  technology  are  appropriate,  now  arid  in  the  future,  for  the  institution  based  on  its 
mission,  culture,  and  use  (Garcia  &  Ratcliff,  1997). 

Future  Use  of  Technology.  Current  technological  capabilities  can  provide  solutions  to  the 
known  assessment  requirements  of  the  faulty  and  staff.  However,  technology  also  enables 
changes  to  future,  anticipated  methods  of  accomplishing  tasks.  This  analysis  of  current 
technology  with  a  look  to  the  future  explores  the  paradigm  that  an  institution  uses  to  view  the 
future  use  of  technology.  One  might  expect  the  advances  in  technology  to  provide  the  stimulus 
to  shift  paradigms  to  include  an  increase  in  electronic  testing. 

In  order  to  accomplish  the  paradigm  shift,  the  technology  must  complement  the  goals, 
mission,  and  student  capabilities.  This  technology  could  include  any  hardware  device  or 
software  required  to  accommodate  the  delivery.  Today  and  continuing  into  the  future,  the 
Internet  and  web-capable  sites  can  deliver  course  materials,  e-mail,  homework  assignments, 
course  instruction,  student-to-student  and  student-to-instructor  interaction,  and  assessments 
(Garcia  and  Ratcliff,  1997;  Bicanich  and  others,  1997).  However,  virtually  all  institutions 
engaging  in  courses  on  the  Internet  have  not  ventured  into  conducting  assessments  using  the 
same  technology  (Burgess,  1994;  Spille  and  others,  1997). 

Sufficient  Technical  Knowledge.  Courses  using  newer  technologies  require  greater 
technical  knowledge  from  the  faculty  and  staff  to  create,  manage,  and  conduct  the  courses. 
Additionally,  students  require  greater  technical  knowledge  to  satisfactorily  participate  and  learn 
during  the  execution  of  the  course.  Unfortunately,  the  literature  documents  a  huge  gap  in 
technical  knowledge  between  the  course  providers  and  course  participants  (Komives  and 
Peterson,  1997;  Engstrom,  1997). 
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More  and  more  faculty  members  are  using  new  technologies  to  perform  tasks  such  as  e- 
mail,  posting  syllabi  on  the  WWW,  administrative  information,  and  the  like.  Institutions  are 
using  these  new  technologies  and  capabilities  to  develop  hundreds  of  undergraduate  and  graduate 
courses  in  the  area  of  distance  education.  New  capabilities  such  as  multimedia  technology 
provide  innovative  delivery  methods  of  courses  to  students  who  are  unable  to  take  courses  in  the 
traditional  way.  In  fact,  this  is  the  latest  boon  to  distance  learning  (Ellington  and  others,  1993). 
Innovative  methods  of  course  delivery  have  enjoyed  huge  successes  touting  a  higher  percentage 
of  students  completing  the  course  than  the  same  course  taught  in  the  traditional  classroom 
(Farmer,  1997;  Gillespie,  1997).  However,  the  course  assessments  are  not  completed  in  the  same 
fashion. 

Students  entering  today's  institutions  have  a  much  higher  level  of  computer  literacy  and 
technical  knowledge  than  previous  generations  (Komives  and  Peterson,  1997).  The  National 
Assessment  of  Educational  Progress  (NAEP)  compiled  data  from  students  in  the  graduating  class 
of  1 996  taking  the  SAT.  They  found  that  72%  of  the  students  had  experience  with  word 
processors  on  computers  and  over  50%  were  labeled  as  computer  literate  (McDaniels,  1997). 
Frequently,  the  students  have  a  better  understanding  of  computers  and  more  knowledge  of 
computing  technology  than  the  average  faculty  member  (Engstrom,  1997).  Consequently, 
students  actively  pursuing  an  academic  education  have  sufficient  technical  knowledge  to  actively 
engage  in  electronic  courses.  However,  each  institution  must  evaluate  its  student  population  to 
determine  the  appropriate  type  of  assessments  and  delivery  methods  (Farmer,  1997). 

Several  studies  provided  substantial  evidence  that  students  are  much  more  active  in  the 
learning  process  when  courses  use  computing  technology.  The  students,  through  the  use  of 
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many  activities,  learn  more  by  collaborating  with  peers  (Engstrom,  1997).  Contrasting  with  this 
benefit,  a  study  conducted  in  1996  indicated  only  38%  of  faculty  used  e-mail  in  their  courses.  In 
addition,  the  faculty  believed  only  3 1%  of  their  students  knew  how  to  use  the  World  Wide  Web 
(WWW)  (Engstrom,  1997).  This  reflects  a  disconnect  between  the  faculty's  perceptions  about 
students'  abilities  and  students’  actual  technological  capabilities. 

To  help  satisfy  technology  requirements  on  campus,  institutions  have  ambitious  plans  to 
dramatically  improve  campus  infrastructures  for  high-speed  telecommunications. 

Unfortunately,  technical  knowledge  and  abilities  of  the  faculty  have  lagged  seriously  behind  the 
technological  advances  (Komives  and  Peterson,  1997;  Bicanich  and  others,  1997). 

Internet  Access  and  Physical  Assessment  Transmission.  For  courses  using  newer 
technologies,  how  to  deliver  materials  becomes  an  issue.  For  an  Internet-based  course,  Internet 
access  by  the  faculty  and  staff  of  the  institution  is  necessary  to  conducting  the  course.  The 
students  involved  in  the  class  must  also  have  access  to  the  same  delivery  method. 

Consequently,  in  addition  to  technical  knowledge,  students  may  also  require  access  to 
personal  computers  and  the  Internet  to  make  the  delivery  of  course  material  and  assessment 
practical.  Not  only  must  institutions  evaluate  what  methods  of  delivery  provide  the  student  with 
best  chance  of  correctly  demonstrating  mastery  of  the  course  material,  innovative  assessments 
should  also  be  considered  (CUSE,  1997;  Hansen,  1994).  Methods  of  assessment  include  paper- 
based  assessments,  group  projects,  individual  projects,  and  experiments.  Each  method  has  its 
own  inherent  advantages  and  disadvantages  associated  with  the  delivery  method  chosen  by  the 
institution  (Kean,  1994;  IDS,  1994). 

Technology  changes  during  the  recent  decade  provided  the  opportunity  to  electronically 
transmit  an  assessment  directly  to  the  student  (Bicanich  and  others,  1997).  This  transmission  can 
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be  interactive  during  the  course  of  the  assessment  or  the  assessment  can  be  transmitted  in  its 
entirety.  Assessments  can  be  transmitted  as  hypertext  from  a  web  page,  e-mail,  telnet,  or 
accessed  via  a  network  (Garcia  and  Ratcliff,  1997).  The  institution  should  consider  these  new 
technological  capabilities  of  assessment  transmission  when  deciding  the  types  of  acceptable 
assessments  and  the  delivery  method. 

Web-Capable  Software.  The  World  Wide  Web  is  providing  faculty  members  unparalleled 
flexibility  in  the  distribution  of  materials.  For  the  first  time,  a  student  can  receive  electronic 
versions  of  syllabi,  assignments,  lectures,  notes,  and  assignments  via  web-capable  software 
accessing  the  institution's  web-site  (Garcia  and  Ratcliff,  1997).  However,  most  faculty  members 
are  not  exploiting  the  capability  but  merely  automating  existing  processes  and  information.  The 
WWW  has  literally  opened  the  institution's  doors  to  students  around  the  world.  Now  institutions 
are  faced  with  decisions  as  to  who  the  target  population  will  be  and  how  the  potential  students 
will  be  best  served.  Also,  today's  students  are  usually  older  than  previous  college  students. 

These  new  students  bring  a  different  view  of  education  and  are  typically  more  mature  and 
motivated  to  learn  (Gubemick  and  Ebeling,  1997). 

Administrative  Issues 

Administrative  issues  are  concerned  with  the  faculty’s  time  and  efforts  to  create, 
maintain,  deliver,  and  score  an  assessment.  Administrative  issues  are  applicable  to  any 
assessment  whether  paper  or  electronic  and  included  such  things  as  faculty  perceptions  of 
computer-based  testing;  ease  of  assessment  development,  maintenance,  and  scoring;  coverage  of 
material;  assessment  difficulty;  efficiency  of  delivery  methods;  and  validity  and  reliability  of 
assessments. 
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Faculty  Perception  of  Computer-Based  Testing.  Research  indicates  that  faculty  tend  to 
equate  computerized  testing  with  training  or  practice  exercises  for  students.  They  also  tend  to 
place  machine  or  automated  scoring  of  answer  sheets  in  the  category  of  computerized  testing 
(Stager  and  Mueller,  1991).  In  addition,  faculty,  as  a  group,  tends  not  to  recognize  the 
possibilities  of  true  computerized  testing.  True  computerized  testing  is  such  that  the  student 
accesses  an  assessment  via  a  computer,  local  or  distant,  and  completes  the  entire  assessment. 
The  assessment  is  scored  by  the  software,  returns  a  score  to  the  student,  and  enters  the 
information  into  an  electronic  database  (Bicanich  and  others,  1997). 


Assessment  Development.  Maintenance,  and  Scoring.  Obviously,  the  choice  of 
assessments  has  implications  for  the  faculty  in  terms  of  what  is  required  to  create,  maintain,  and 
score  the  selected  assessments.  Electronic  assessments,  while  requiring  the  effort  to  create  and 
maintain  a  suitable  set  of  instruments,  can  provide  real  advantages  to  the  faculty. 

Table  2  lists  several  potential  benefits  of  computerized  testing  (Stager  and  Mueller, 
1991).  The  original  table  was  reduced  to  the  relevant  items  for  this  research. 

Table  2.  Benefits  of  Computerized  Testing 


Automates  the  process  of  creating  tests _ 

Automates  the  process  of  scoring  tests 

Facilitates  the  creation  of  equivalent  versions  of  the  same  exam 

Provides  access  to  existing  test  banks _ 

Standardizes  test  administration  procedures _ 

Provides  more  detailed  feedback  to  the  student _ 

Enables  teachers  to  become  classroom  researchers _ 

Provides  analysis  of  the  errors  in  strategies  students  are  using 

(From  Stager  and  Mueller,  1991:248) 
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According  to  Stager  and  Mueller,  institutions  should  evaluate  each  of  these  benefits 
based  on  the  culture  of  that  institution.  There  are  software  packages  commercially  available  to 
provide  all  of  the  benefits  listed  in  the  table.  More  importantly,  not  only  the  faculty  but  students 
can  benefit  from  computerized  testing.  In  addition,  software  can  automate  the  process  of 
creating  equivalent  assessments.  Equivalent  exams  are  critical  to  ensure  that  each  student 
receives  an  assessment  equally  as  difficult  as  any  other  student.  Producing  assessments  of  equal 
difficulty  is  particularly  important  when  randomly  generating  assessments  (Stager  and  Mueller, 
1997). 

To  enhance  computer-based  testing,  some  institutions  have  used  test  banks  to  house  valid 
and  reliable  sets  of  assessment  items.  The  test  banks  provide  an  automated  method  of 
controlling,  updating,  and  using  the  available  information.  By  automating  the  assessments,  all  of 
the  students,  whether  they  are  the  first  class  or  tenth  class,  receive  identical  instructions.  In 
traditional  assessment  delivery,  groups  of  students  can  receive  very  different  instructions.  Also, 
faculty  may  inadvertently  give  one  group  an  advantage  by  providing  additional  information 
(Stager  and  Mueller,  1991).  For  example,  as  the  instructor  administers  the  exam  to  successive 
classes,  the  instructions  given  to  the  students  will  change. 

Research  also  suggests  institutions  should  be  interested  in  the  requirements  needed  to 
create,  maintain,  and  score  the  assessment  (Bicanich  and  others,  1 997).  Computers  are  pervasive 
in  institutions  today  and  are  normally  used  by  faculty  to  automate  the  creation  of  assessments  by 
using  word  processing  packages.  However,  institutions  have  underutilized  the  potential  benefits 
of  automating  the  delivery  of  the  assessments.  Once  the  assessment  is  created,  it  is  much  easier 
to  maintain  the  assessment  in  the  electronic  form  than  a  traditional  paper-based  examination. 

Ease  of  scoring  is  also  a  primary  consideration  of  faculty  particularly  when  many  students  take 
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the  examination  (Brigance  and  Hargis,  1993;  CUSE,  1997).  When  the  student  has  completed  the 
assessment,  the  software  records  all  of  the  desired  information  concerning  the  assessment  in  a 
database. 

Computers  can  also  provide  interactive  assessments  and  can  handle  a  far  greater  number 
of  students.  Whether  using  a  multiple-choice,  essay,  or  other  forms  of  assessments,  new 
technologies  can  assist  the  faculty  in  performing  the  evaluation  of  students  (Wubbels  and  Girgus, 
1997;  Hansen,  1994).  Ease  of  scoring  can  dramatically  reduce  the  effort  required  by  faculty 
while  maintaining  the  ability  to  have  direct  knowledge  of  a  specific  student's  performance 
(CUSE,  1997). 

Course  Material  Coverage.  All  assessments,  independent  of  type  or  delivery  method,  must 
provide  adequate  coverage  of  the  course  material.  The  curriculum  content  is  derived  first  and 
then  the  assessment.  The  assessment  should  contain  an  adequate  sample  of  the  course  material 
deemed  important  for  mastery  (Brigance  and  Hargis,  1993;  CUSE,  1997).  In  addition,  the 
assessment  must  be  valid  and  reliable  by  clearly  evaluating  the  correct  course  material  at  a 
difficulty  level  commensurate  with  the  institution's  policies. 

Another  administrative  aspect  for  an  assessment  should  be  the  timely  measurement  of 
student  performance  that  allows  feedback  for  the  student.  A  goal  of  the  assessment  should  be  to 
gain  insight  into  students'  learning  or  curriculum  deficiencies  (Brigance  and  Hargis,  1993).  One 
or  more  students  may  not  be  learning  the  material  as  desired  and  consequently  the  assessment 
should  provide  that  information  to  the  faculty.  In  addition,  curriculum  changes  may  be  in  order, 
but  the  faculty  member  may  not  realize  this  if  assessments  are  simply  used  as  part  of  the  course 
grade. 
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To  gain  more  insight  into  the  student  and  course  performance,  conducting  automated 
assessments  allows  faculty  members  to  become  classroom  researchers  (Stager  and  Mueller, 
1991).  Assessments  can  not  only  provide  basic  information  such  as  scores,  number  right  or 
wrong,  but  also  in-depth  information.  For  example,  statistics  such  as  which  items  were  missed, 
frequency  of  items  missed,  and  item  distracters  selected  most  often  can  readily  be  made  available 
to  the  instructor.  The  statistics  can  provide  useful  information  to  the  instructor.  For  example, 
knowing  that  a  particular  distracter  is  being  selected  at  a  high  rate  may  indicate  a  flaw  in  course 
delivery  (Stager  and  Mueller,  1991). 

Finally,  computerized  testing  can  provide  insight  into  a  student’s  mistakes.  For  example, 
if  the  question  were  "What  is  34  +  4?"  If  the  student  selected  the  distracter  with  30  as  the 
answer,  this  would  seem  to  indicate  the  student  mistakenly  subtracted  instead  of  added.  If  the 
student  selected  the  distracter  with  83  as  the  answer,  the  student  might  be  inverting  the  number 
38  (Stager  and  Mueller,  1991 :  259).  Consequently,  a  well-developed  set  of  distracters  written 
within  the  context  of  a  question  could  have  inherent  course  and  student  evaluation  information. 

Assessment  Difficulty.  Assessment  difficulty  refers  to  the  ability  of  the  assessment  to 
determine  the  level  of  knowledge  achieved  by  the  student.  The  assessment  is  meant  to  provide 
the  instructor  with  sufficient  reassurance  that  the  student  did  in  fact  learn  the  material. 

Electronic  assessments  should  be  sufficiently  difficult  to  reach  the  level  of  domain  coverage  as 
required  by  the  institution  (Cizek,  1997). 

Table  3  displays  common  assessment  approaches  as  they  relate  to  the  achievement  targets 
of  the  course.  Within  content  acquisition,  there  are  two  subdivisions:  select-type  and  supply-type 
formats.  The  essential  difference  is  select-type  is  more  indicative  of  recognition  memory  as 
opposed  to  supply-type  requiring  more  recall  memory.  Procedural  knowledge  and  performance 
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both  require  a  great  deal  of  recognition  and  recali  memory  (CUSE,  1997).  These  last  two  targets 
are  gaining  favor  in  institutions  in  the  form  of  individual  or  group  projects.  Some  institutions  are 
assessing  student's  mastery  through  projects  requiring  the  use  of  skills  learned  in  the  course  and 
the  student's  own  creativity. 


Table  3.  Taxonomy  of  Assessment  Approaches 


Achievement  Targets 

Content  Acquisition 

Select-type  formats  (e.g.,  multiple  choice,  true/false,  alternate 
choice,  matching) 

Supply-type  formats  (e.g.,  short  answer,  fill  in  the  blank,  label  a 
diagram) 

Procedural  Knowledge 

Describe  a  process  (e.g.,  lab  experiment,  operate  a  machine, 

construct  a  flowchart,  direct  observation,  "show 
your  work",  "tell  the  steps  you  followed") 

Performance 

Demonstrations  (e.g.,  build  a  birdhouse,  repair  a  car,  write  an  essay, 
debate  an  opponent,  recite  a  poem,  compose  a  song, 
lead  a  discussion,  compete  in  an  event,  create  a 
sculpture) 

(Cizek,  1997:22) 


Delivery  Methods.  Similar  to  traditional  assessments,  electronic  assessments  must  be 
understandable  in  that  the  student  must  not  get  lost  in  the  technology  of  the  assessment  delivery. 
An  electronic  assessment  requires  clarity  when  soliciting  a  response  from  a  student.  Some 
faculty  are  concerned  that  the  electronic  delivery  of  assessments  is  inherently  inferior  to  paper- 
based  assessments.  However,  at  least  one  study  found  Internet-delivered  and  paper-based 
assessments  to  be  equivalent  using  the  test-retest  design  (Bicanich  and  others,  1997). 


Validity  and  Reliability.  An  assessment  that  measures  what  it  was  developed  to  measure  has 
high  validity.  One  study  indicates  the  key  to  Internet  testing  is  having  high  test  validity 
(Bicanich  and  others,  1997).  An  important  type  of  validity  is  content  validity.  An  assessment 
with  high  content  validity  confirms  that  the  instructor  is  indeed  measuring  the  course  material  the 
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student  received  during  the  course.  Consequently,  researchers  argue  that  the  assessment  should 
be  explicitly  derived  from  the  important  portions  of  the  course  and  cover  the  domain  of 
knowledge  for  the  course  (Brigance  and  Hargis,  1993;  Ellington,  1993;  Jacobs  and  Chase,  1992). 

Assessments  providing  consistent  results  from  repeated  administrations  to  the  same  or 
similar  groups  have  high  reliability  (Ellington,  1993;  Jacobs  and  Chase,  1992).  There  are  three 
major  types  of  assessment  reliability:  test-retest  reliability,  internal  reliability,  and  interrater 
reliability  (Brigance  and  Hargis,  1993:65).  Test-retest  reliability  is  concerned  with  correlating 
the  scores  on  several  iterations  of  the  assessment  to  ensure  the  instrument  yields  similar  results. 
Internal  reliability  is  concerned  with  having  an  assessment  where  all  of  the  items  are  at  the  same 
level  of  difficulty  and  are  measuring  the  same  domain  of  knowledge.  Interrater  reliability  is  a 
reflection  of  the  ability  of  the  instrument  to  produce  similar  results  from  different  evaluators. 
Each  of  these  types  of  reliability  is  critical  for  electronic  assessments.  For  example,  a  randomly 
generated,  multiple-choice  assessment  would  be  reliable  if  the  software  is  constructed  to  ensure 
that  each  randomly  generated  examination  is  equivalent  and  repeated  executions  yield  similar 
results. 

Inertia 

Inertia,  as  used  in  this  study,  is  the  tendency  of  an  institution  to  resist  changes  and 
maintain  the  status  quo.  This  issue  also  encompasses  the  institution's  concern  with  enhancing  or 
maintaining  its  academic  reputation.  This  research  focuses  on  the  resistance  to  using  new 
technology  and  to  providing  electronic  assessments.  For  example,  Dr.  Updogrove  of  Yale 
University  indicated  many  of  Yale's  professors  doubt  distance  education  courses  can  meet  the 
"quality  dimensions  that  make  Yale  comfortable"  (Blumenstyk,  1997).  Additionally,  James 
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Aisner  of  Harvard  Business  School  indicates  Harvard  also  has  a  tendency  to  resistance  change. 
Mr.  Aisner  says  that  distance  education  "goes  against  what  Harvard  stands  for  in  terms  of  the 
learning  process"  (Gubemick  and  Ebeling,  1997).  Seen  in  this  light,  inertia  can  be  a  stabilizing 
influence,  one  that  protects  the  quality  and  heritage  of  an  institution.  On  the  other  hand,  inertia 
can  manifest  itself  in  such  things  as  resistance  to  pedagogy  changes,  over  sensitivity  to  perceived 
reputation,  a  lack  of  strategic  vision  about  the  future  of  developing  and  using  assessments,  as 
well  as  concerns  about  maintaining  academic  rigor  and  standards. 

Pedagogy  Changes.  Institutions  in  the  academic  arena  tend  to  move  slowly  when  facing 
changes  to  the  curriculum  (Stager  and  Mueller,  1991;  Mojkowski,  1990).  The  faculty  typically 
resists  efforts  to  introduce  pedagogical  changes  whether  these  changes  originate  from  the 
institution’s  leadership  or  from  new  technological  capabilities. 

Stager  and  Mueller  argue  that  education  professionals  believe  that  computerized  testing  is 
inferior  to  paper-based  testing.  They  list  eight  criticisms  of  computerized  testing.  Table  4  lists 
the  six  criticisms  that  are  applicable  to  this  research. 

Table  4.  Criticisms  of  Computerized  Testing 

Is  expensive 

Produces  lower  student  achievement 
Is  limited  to  assessing  information  recall  and  recognition 
Separates  the  student  evaluation  process  from  instruction 
Increases  student  anxiety  about  the  testing  process 

Removes  the  teacher  from  the  evaluation  process _ 

(From  Stager  and  Mueller,  1991:260; 

Although  when  Stager  and  Mueller  wrote  this  article,  computing  technology  was  more 
expensive,  institutions  have  since  spent  millions  of  dollars  to  provide  sufficient  computing  and 
communication  capabilities  for  activities  other  than  for  assessments.  Consequently,  the 
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additional  cost  of  computerized  testing  would  be  relatively  inexpensive  (Bicanich  and  others, 
1997). 

Similarly,  research  had  shown  that  students  performed  equally  as  well  between  paper- 
based  and  computerized  assessments.  However,  a  form  of  computerized  testing  known  as 
adaptive  testing  caused  suspicion  in  the  minds  of  some  (Morrow,  1997).  The  GRE  and  GMAT 
are  assessments  that  use  this  methodology  (GMAC,  1998;  GRE,  1998).  This  form  of  assessment 
determines  the  next  question  based  on  the  student’s  performance  on  previous  questions.  Not  all 
computerized  assessments  use  the  adaptive  style  of  testing. 

Another  criticism  is  that  faculty  perceives  computer-based  testing  is  limited  to  testing 
recall  or  recognition  memory.  Research  has  shown  that  this  is  a  problem  with  assessments  as  a 
whole  and  is  not  limited  to  computerized  testing.  Research  shows  that  80%  of  all  paper-based 
assessments  are  limited  to  testing  recall  or  recognition  memory  (Stager  and  Mueller,  1991). 

The  criticism  that  students  feel  anxiety  when  taking  a  test  on  a  computer  is  simply  not 
substantiated  by  research.  At  one  institution  97%  of  the  students  reported  that  they  liked 
computerized  tests  "a  lot"  and  at  a  second  institution  68%  of  the  students  preferred  computerized 
tests  (Stager  and  Mueller,  1991). 

The  criticism  that  faculty  members  may  become  detached  from  the  assessment  process 
has  some  merit.  Some  faculty  members  indicated  that  they  like  the  interaction  of  delivering  an 
assessment  and  then  personally  grading  the  assessment  in  order  to  make  a  better  evaluation  of  a 
student’s  abilities  (Hansen,  1994;  Kean,  1994). 

Besides  being  critical  of  computerized  testing,  many  faculty  members  have  been  slow  to 
use  technology  to  improve  pedagogy.  In  many  institutions  today,  faculty  have  Internet  access 
directly  from  their  desk  computers.  Typically,  this  access  to  the  Internet  does  not  result  in  the 
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faculty  incorporating  this  capability  into  curricula  (Farmer,  1997).  One  reason  for  the  failure  to 
incorporate  the  Internet  into  curricula  is  that  computer  technology  is  advancing  much  more 
quickly  than  academic  institutions  can  update  their  pedagogy.  Charles  Mojkowski  posed  the 
following  three  questions  for  institutions  to  evaluate  concerning  new  technology  and  the 
educational  process  (Mojkowski,  1990:13): 

1 .  How  should  existing  curriculum  and  pedagogy  be  redesigned  to 
accommodate  and  maximize  the  use  of  the  technology? 

2.  What  are  the  most  appropriate  ways  to  bring  technology  into  the 
curriculum? 

3.  How  can  computers  and  other  new  technological  tools  be  used  as 
catalysts  to  revitalize  existing  curriculum  and  instruction  within  and  across 
subject  areas? 

Mojkowski's  perspective  is  that  technology  is  not  providing  the  motivation  or  capability 
to  provide  improvements  to  educational  curricula  but  is  simply  being  used  to  automate  existing 
processes.  Institutions  are  spending  millions  of  dollars  annually  on  computer  technology  without 
regard  to  the  improvements  in  pedagogy  this  technology  can  provide  (Mojkowski,  1990;  White, 
1990).  However,  the  research  revealed  that  many  institutions  have  adopted  new  computing 
technologies  to  deliver  distance  education  courses.  Some  institutions  are  using  the  Internet  to 
deliver  many  courses  in  an  asynchronous  fashion  and  charging  a  premium  for  that  service.  Many 
of  the  same  institutions  also  require  paper-based  examinations  for  the  courses  as  opposed  to 
using  an  assessment  commensurate  with  the  course  delivery  method.  Others,  such  as  Duke 
University's  distance  education  program,  use  group  projects  as  a  method  of  student  assessment. 
For  an  Internet  course,  for  example,  the  assessment  might  be  a  web-based,  electronic  project. 

Stager  and  Mueller  state  the  reason  computer  technology  has  not  been  incorporated  into 
education  pedagogy  is  training  for  faculty  tends  to  focus  on  computer  literacy,  using  computers 
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to  expedite  personal  productivity,  and  the  actual  mechanics  of  computing  (Stager  and  Mueller, 
1991).  In  addition,  instructors  tend  to  teach  in  a  method  similar  to  the  way  they  were  taught 
which  was  without  computers  (Stager  and  Mueller,  1997).  However,  using  computers  to 
enhance  productivity  does  not  necessarily  create  pedagogical  changes.  When  it  does  not,  then 
technology  is  not  being  used  to  the  maximum  extent  possible  (Engstrom,  1997).  Therefore,  a 
pedagogical  change  must  occur  from  within  the  institution  to  produce  support  for  a  paradigm 
change  that  includes  conducting  assessments  commensurate  with  the  delivery  method  of  the 
course.  The  pedagogical  change  will  require  a  shift  in  the  institution's  inertia  before  the  actual 
change  can  occur,  but  the  change  will  necessitate  sufficient  knowledge  of  the  technology  and  the 
willingness  to  use  it. 

Computers  offer  new  and  potentially  valuable  methods  of  delivering  the  assessment  to 
students.  This  area  offers  institutions  the  opportunity  to  address  the  "underdeveloped  pedagogy" 
of  electronic  examinations  (Wubbels  and  Girgus,  1997:280).  Additionally,  computers  can 
provide  interactive  assessments  and  can  handle  a  far  greater  number  of  students.  Whether  using  a 
multiple-choice,  essay,  or  other  forms  of  assessments,  new  technologies  can  assist  the  faculty  in 
performing  the  assessment  evaluation  of  students  (CUSE,  1997). 

In  1990,  a  cooperative  venture  between  the  American  Federation  of  Teachers,  the 
National  Council  on  Measurement  in  Education,  and  the  National  Education  Association 
developed  a  set  of  essential  standards  for  faculty  competence  in  educational  assessments.  The 
standards  were  intended  to  provide  guidance  in  the  training  of  educators,  accreditation  of 
preparation  programs,  and  the  certification  of  future  educators.  The  minimum  standards  of 
faculty  members  are  shown  in  Table  5. 
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Table  5.  Faculty  Member  Assessment  Knowledge  Standards 


Skilled  in  choosing  assessment  types _ 

Skilled  in  developing  assessments _ 

Skilled  in  administering,  scoring,  and  interpreting  results _ 

Skilled  in  using  assessment  results  when  making  decisions _ 

Skilled  in  developing  valid  student  grading  procedures  that 
Use  student  assessments 

Skilled  in  communicating  results  to  students,  parents,  others 
Skilled  in  recognizing  unethical,  illegal,  and  other  inappropriate 

Assessment  methods  and  uses  of  assessment  information _ 

(Phye,  1997:36 1 


Reputation.  The  literature  suggests  that  all  institutions  are  sensitive  to  their  perceived 
academic  reputations.  The  literature  does  not  specifically  mention  assessments  as  contributing  to 
or  detracting  from  an  institution's  reputation,  but  it  is  implied  nonetheless.  Jacobi  discusses  two 
common  approaches  to  define  academic  excellence:  reputation  and  resource.  Reputation  is 
equated  to  the  institution's  perceived  excellence  and  prestige  in  the  academic  world.  Resource 
relates  to  comparisons  based  on  criteria  such  as  faculty  productivity,  endowments,  or  assessment 
scores.  The  results  from  the  two  approaches  tend  to  reinforce  one  another.  An  increase  in 
reputation  can  increase  resources  and  likewise  an  increase  in  resources  can  increase  the 
reputation  (Jacobi  and  others,  1987). 

Strategic  Vision.  The  literature  supports  the  notion  that  all  assessments  should  be  used  for 
more  than  simply  arriving  at  grades  for  students.  Assessments  should  be  developed  to  provide 
insight  into  whether  goals  and  objectives  are  being  met  (Jacobi  and  others,  1987).  Although 
some  of  the  literature  focuses  on  institution-wide  assessments,  each  faculty  member  should 
incorporate  the  course  goals  and  objectives  into  each  course  assessment.  Consequently, 
institutions  should  have  a  strategic  perspective  in  evaluating  future  forms  of  assessments. 
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Institutions  should  address  early  in  the  course  development  process  what  the  appropriate 
course  material  and  the  corresponding  assessment  will  be  for  each  course.  Entrance  and  exit 
criteria  must  be  clearly  matched  to  the  goals  and  desired  behavior  of  the  students.  Therefore, 
assessments  should  meet  the  institution's  standards,  which  in  turn  are  carefully  chosen  to  reflect 
the  goals  the  institution  is  striving  to  achieve  (Doherty  and  others,  1997).  The  issue  of  standards 
relates  directly  to  the  institution's  strategic  goal  for  the  overall  institution.  If  an  institution  has  a 
long-term  goal  to  increase  academic  standards,  one  issue  requiring  attention  is  the  acceptable 
form  of  assessment  (Jacobi  and  others,  1987;  Kean,  1994). 

Academic  Rigor.  Assessments  should  be  of  sufficient  academic  rigor  to  ensure  the  course 
material  was  in  fact  learned  by  the  student  (Hansen,  1994).  Also,  assessments  should  be 
accomplished  in  a  manner  similar  to  how  the  student  will  use  the  information  (Doherty  and 
others,  1997).  For  example,  collaborative  learning  is  used  extensively  in  subject  areas  where  the 
student  will,  upon  graduation,  use  the  information  in  a  team-oriented  process.  Institutions  also 
use  assessments  to  enhance  their  reputation  and  standing  among  their  peers.  Schools  that  are 
among  the  most  academically  stringent  are  also  among  the  highest  ranked  academic  institutions 
(Thompson  and  Morse,  1 997). 

Standards.  All  academic  assessments,  of  any  form,  are  normally  of  two  types:  criterion- 
referenced  and  norm-referenced  (Spencer,  1991;  Hansen,  1994;  CUSE,  1997).  Criterion- 
referenced  assessments  judge  a  student's  performance  against  an  established  standard.  The 
standard  is  determined  for  a  course  prior  to  the  execution  of  the  assessment.  Criterion- 
referenced  assessments  also  have  minimum  acceptable  performance  standards  inherent  in  the 
assessment.  A  student  will  simply  meet  or  fail  to  meet  the  standard  for  the  assessment.  Norm- 
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referenced  assessments  judge  the  students  in  relation  to  each  other.  Students'  success  or  failure 
is  not  related  to  a  minimum  acceptable  standard.  Either  type  can  be  applied  to  electronic 
assessments.  However,  the  assessment  must  be  consistent  with  institution  standards  (Kean, 
1994). 

Institutional  Politics.  Institutions  are  not  immune  to  the  phenomena  of  inertia.  Inertia  can 
exhibit  itself  by  the  reluctance  to  affect  pedagogy  changes,  long-term  direction,  or  future  actions 
unless  highly  encouraged  to  do  so.  In  fact,  institutional  politics  can  be  such  that  research 
outcomes  advocating  a  change  in  policy  or  program  will  be  criticized  based  on  research 
methodology  as  opposed  to  the  findings  themselves  (Jacobi  and  others,  1987).  The  following 
quote  exemplifies  this  attitude: 

Further,  many  post  secondary  institutions  are  highly  conservative  and  faculty 
or  administrators  may  be  vested  in  maintaining  status  quo.  Under  such 
circumstances,  resistance  is  mobilized  when  change  is  recommended,  and 
information  about  outcomes  may  become  a  victim  of  academic  gamesmanship. 

(Jacobi  and  others,  1987:1 1) 

One  institution  created  workshops  to  help  the  faculty  learn  how  to  change  the  focus  from 
teaching  to  student  learning.  The  faculty  resisted  this  effort  and  asserted  that  the  new  focus 
violated  their  academic  freedom  to  conduct  courses  in  the  manner  they  saw  fit.  Also,  the  faculty 
indicated  the  changes  placed  too  much  attention  on  students  and  reduced  the  faculty's  role  as  the 
center  of  the  learning  process  (Komives  and  Petersen,  1997;  Gubemick  and  Ebeling,  1997). 

Security  Issues 

Security  issues  are  concerned  with  the  institution’s  ability  to  provide  a  safe  and  secure 
environment  for  the  creation,  storage,  delivery,  and  scoring  of  an  assessment.  Security  issues 
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include  such  things  as  configuration  of  software  and  hardware  to  enforce  security;  access 
controls;  and  use  of  proctors  to  minimize  cheating. 

Security  Software  and  Hardware  Configurations.  Many  software  products  and  hardware 
configurations  are  available  for  the  institution  to  effectively  secure  electronic  assessments.  Most 
institutions  have,  at  a  minimum,  rudimentary  implementations  of  these  capabilities  currently  in 
place.  They  may  simply  need  to  enhance  these  capabilities  to  ensure  a  level  of  security 
commensurate  with  the  standards  the  institution  is  determined  to  achieve  (Bicanich  and  others, 
1997). 

Password/Restrict  Access.  Obviously,  assessments  require  protection  from  unauthorized 
access.  Access  controls  must  be  sufficient  to  allow  only  legitimate  students  enrolled  in  the 
course  to  gain  access  to  the  assessment.  Several  methods  exist  to  help  ensure  that  access  control 
is  maintained.  The  most  basic  and  relatively  effective  method  is  to  restrict  access  by  assigning 
each  student  a  unique  login  and  password  (Bicanich  and  others,  1997).  This  method  places  the 
security  of  students'  logins  and  passwords  in  the  students'  hands  to  prevent  compromise. 

Assessment  Security.  Assessment  software  is  also  available  to  provide  additional  security 
by  allowing  the  test  to  be  taken  only  during  certain  hours,  limiting  the  number  of  times  the 
student  can  access  the  assessment,  accurately  recording  answers,  etc.  In  addition,  many 
operating  systems  provide  extensive  tracking  of  logins  and  recording  of  system  activities  related 
to  that  login  (Bicanich  and  others,  1997). 

Physical  Environment.  One  institution  conducted  an  experiment  in  1988  using  an  older 
mainframe  computer  with  terminals  housed  within  a  single  room.  The  students  were  allowed  to 
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take  the  required  assessments  at  their  convenience  during  a  predetermined  period  of  time.  There 
were  two  noteworthy  comments  pertaining  to  the  research.  First,  the  students  believed  the 
amount  of  cheating  was  equal  to  other  tests  but  they  perceived  additional  opportunities  to  cheat. 
Secondly,  following  the  course,  70%  of  the  students  preferred  the  computer  tests  to  paper-based 
tests  (Gwinn  and  Beal,  1988).  A  more  recent  study  found  68.5%  of  the  360  participating  high 
school  students  had  little  to  no  prior  Internet  experience.  Nevertheless,  the  students  favored  the 
web-based,  electronic  assessment  by  a  3-to-l  margin  over  the  paper-based  equivalent  assessment 
(Bicanich  and  others,  1997). 

Cheating.  Another  area  of  concern  in  the  administrative  realm  in  education  today  is  the  use 
of  proctors  as  the  best  safeguard  against  possible  cheating  by  a  student.  The  literature 
exemplifies  this  idea  by  the  following  quote:  "Everyone  knows  that  some  students  cheat,  and 
some  believe  that  eveiyone  cheats  some  time"  (Aiken,  1991:  726).  Aiken  suggests  increasing 
the  number  of  proctors  as  the  best  way  to  prevent  cheating.  According  to  a  different  study  in 
1981,  48%  of  college  faculty  members  believed  proctors  watch  examinees  consistently. 

However,  only  21%  of  the  students  agreed  (Barnett  and  Dalton,  1981).  In  addition,  there  are 
software  packages  available  that  could  reduce  cheating  without  increasing  the  number  of 
proctors.  Such  software  uses  statistical  modeling  to  establish  the  probabilities  of  two  students 
answering  exactly  the  same  questions  on  the  test.  For  example,  if  one  student  missed  the  same 
questions  as  another  student  and  they  both  missed  them  by  selecting  the  same  distracters,  the 
software  will  establish  the  probability  that  this  could  occur  (Aiken,  1991). 

Traditional  measures  have  centered  on  preventing  cheating  as  opposed  to  trying  to  catch 
cheating  after  the  fact  (Aiken,  1991).  Methods  include  multiple  exams,  random  seating 
arrangement,  placing  empty  seats  between  students,  and,  of  course,  multiple  proctors.  One  study 
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cited  by  Aiken  found  exams  with  rearranged  questions  and  answers  effectively  eliminated 
detectable  cheating  (Aiken,  1991).  A  supporting  study  found  scrambling  the  exams  and/or  the 
seating  arrangement  virtually  eliminated  cheating  (Harpp  and  others,  1996).  One  study  found 
that  the  larger  the  portion  of  students  living  on  campus,  the  less  likelihood  the  students  would 
cheat  (McCabe  and  Trevino,  1996).  In  a  study  using  electronic  assessments,  cheating  was 
reduced  an  estimated  four  fold  by  informing  the  students  that  software  would  be  used  to  analyze 
their  answers  to  detect  cheating  (Bellezza  and  Bellezza,  1989). 

Other  ways  to  minimize  cheating  include  using  randomly  generated  assessments  so  that 
no  two  students  receive  the  same  items  and  creating  the  assessment  such  that  it  does  not  contain 
items  requiring  simple  look-ups  in  the  textbook  (CUSE,  1997). 

Cost  Issues 

An  institution  should  consider  the  costs  associated  with  the  assessment  and  the  facets 
required  to  provide  the  assessment  to  the  students  (Hansen,  1 994).  Equipment  required  to 
conduct  various  types  of  assessments  could  be  vastly  different.  Consider  the  differences  between 
conducting  a  simple  paper-based,  multiple-choice  assessment  and  implementing  a  randomly 
generated,  multiple-choice  assessment  delivered  via  the  Internet.  The  technology  and  knowledge 
required  to  undertake  either  type  should  be  a  factor  considered  by  the  institution  (Engstrom, 
1997).  Assessment  costs  includes  such  things  as  delivery  costs,  hardware  and  software  costs, 
and  assessment  development  and  maintenance  costs. 

Delivery  Costs.  Assessment  delivery  costs  for  the  selected  assessment  can  be  minimal  or 
expensive  (Farmer,  1997).  For  example,  a  paper-based  assessment  delivered  during  a  class 
period  by  the  instructor  has  virtually  no  delivery  costs.  On  the  other  end  of  the  spectrum,  a  web- 
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based,  electronic  assessment  delivered  via  the  Internet  has  much  higher  costs  associated  with  the 
delivery  method.  However,  after  the  initial  installation  of  the  computing  and  communications 
capabilities,  the  delivery  cost  of  each  subsequent  assessment  is  virtually  zero. 

For  paper-based  exams  in  distance  education  courses,  there  are  substantial  postage  and 
handling  costs.  There  are  time  and  salary  considerations  for  the  staff  to  prepare  the  exam  for 
mailing  and  the  receipt  of  the  completed  exam.  Another  consideration  is  the  cost  associated  with 
providing  a  proctor. 

Hardware  and  Software  Costs.  The  costs  for  computing  hardware  and  software  to  house, 
deliver,  score,  and  store  results  for  the  assessment  also  need  to  be  considered.  Additionally,  an 
institution  may  require  higher  quality  communications  capabilities  to  deliver  this  type  of 
assessment.  If  the  hardware  and  software  are  purchased  specifically  for  the  assessment  effort, 
then  the  costs  are  higher.  However,  today's  institutions  have  extensive  computing  capabilities 
and  most  likely  will  simply  purchase  the  software  required  to  properly  process  the  assessment, 
thus  lowering  the  direct  costs  attributed  to  a  specific  assessment  (Bicanich  and  others,  1997). 

Assessment  Development  and  Maintenance  Costs.  Assessment  development  costs  add 
a  further  dimension  when  considering  the  various  forms  an  assessment  can  take.  Costs  for 
developing  an  assessment  also  reflect  a  range  depending  on  the  selected  type  of  assessment.  The 
faculty  with  a  limited  expenditure  of  effort  normally  accomplishes  paper-based,  traditional 
assessments.  Routinely,  a  word  processor  is  used  to  develop  several  assessments  and  each  is 
used  alternatively  over  time.  However,  one  must  also  include  assessment  maintenance  and 
scoring  costs  in  with  the  development  costs.  Some  assessments  require  more  time  to  score  than 
do  others.  The  obvious  comparison  is  between  the  multiple-choice  and  essay  assessments. 
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Assessment  maintenance  is  the  act  of  updating  the  assessment  to  reflect  changes  in  the 
material  or  focus  of  the  course.  For  an  automated  examination,  the  process  is  similar.  However, 
special  software  is  required.  In  addition,  any  software  required  to  score  the  assessment  and  then 
store  the  results  incurs  costs.  In  a  study  conducted  among  Pennsylvania  high  schools,  software 
required  to  create,  disseminate,  score,  and  store  results  had  a  break-even  point  at  375  students 
(Bicanich  and  others,  1997). 

Assessment  Selection  Model  (ASM) 

Th  ASM  was  constructed  by  this  researcher  based  on  the  preceding  review  of  the 
literature  and  to  help  organize  thinking  around  these  issues.  This  model  provides  a  foundation 
for  identifying  the  issues  that  universities  find  germane  to  the  assessment  selection  process. 
Additionally,  this  model  demonstrates  the  issues’  relative  influence  upon  a  university’s 
assessment  selection  tendencies. 

The  Assessment  Selection  Model  is  comprised  of  five  factors:  technology,  cost,  security, 
administration,  and  inertia.  Inertia  is  central  to  the  model  in  that  the  remaining  constructs 
interact  individually  and  collectively  either  with  or  against  inertia.  In  other  words,  inertia  is  the 
institution’s  resistance  to  change,  but  the  other  constructs  cause  a  decrease  or  increase  in  that 
inertia  to  prevent  or  allow  the  movement  toward  a  different  assessment.  Table  1  at  the  beginning 
of  this  chapter  reveals  how  the  individual  issues  aligned  with  their  respective  factors. 

The  individual  issues  are  the  terms  that  are  expected  to  be  used  by  institutions  as  they 
describe  the  factors  that  they  considered  in  the  assessment  selection  process.  The  ASM  can  then 
be  used  to  classify  the  responses  according  to  the  mapping  scheme.  Figure  1  pictorially 
represents  the  relationships. 
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Figure  1.  Assessment  Selection  Model 


The  Assessment  Selection  Model  represents  the  institution’s  inertia  as  the  current  state  of 
institutional  assessment  practices.  This  is  the  tendency  to  do  things  as  they  always  have  been 
done.  Administrative  and  security  issues  will  likely  reinforce  the  institution’s  inertia.  The 
faculty  has  a  preferred  method  of  developing,  conducting,  and  scoring  assessments  and  does  not 
desire  to  deviate  from  that  course.  Security  will  tend  to  reinforce  the  status  quo  because  the 
current  level  of  security  and  level  of  threat  are  known,  understood,  and  minimized  in  the 
faculty’s  mind.  New  assessments  may  create  unknown  threats  or  require  activities  not  currently 
performed.  Based  on  this,  the  inertia  of  the  institution  will  remain  with  the  current  form  of 
assessments. 
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Costs  and  technology,  with  sufficient  force,  do  not  reinforce  but  alter  inertia.  Increasing 
costs  of  current  practices  may  tend  to  reduce  the  use  of  current  assessments  particularly  if  new 
technologies  offer  opportunities  to  do  assessments  for  less.  If  costs  become  prohibitively 
expensive,  an  institution  will  most  likely  alter  the  status  quo.  Reductions  in  support  staff  or 
faculty  could  require  a  streamlining  effort  to  move  the  institution  toward  technology  enablers. 
Automated  assessments  have  lower  long-term  manpower  requirements  for  maintenance  and 
distribution  and  might  therefore,  provide  an  alternative  previously  not  desired. 

Research  Propositions 

In  order  to  address  the  research  questions,  five  research  propositions  are  developed 

below: 

Proposition  1 

Security  initiatives  and  capabilities  can  provide  a  known,  safe  environment  for  the 
creation,  storage,  delivery,  and  scoring  of  an  assessment.  Frequently,  the  institution's  standards 
and  reputation  will  drive  the  level  of  security  desired  by  that  institution  (Bicanich  and  others, 
1997).  Access  controls  used  to  ensure  only  authorized  students  are  allowed  to  gain  access  to  the 
assessment  are  normally  implemented  with  a  unique  login  and  password  (Bicanich  and  others, 
1997). 

Another  area  of  concern  in  education  is  the  possibility  of  student  cheating.  Security 
measures  must  be  in-place  to  ensure  the  institution's  standards  of  conduct  are  achieved.  There 
are  many  software  packages  available  using  statistical  modeling  to  establish  the  probabilities  of 
two  students  answering  exactly  the  same  questions  on  an  exam.  In  a  study  using  electronic 
assessments,  cheating  was  reduced  four  fold  by  simply  informing  the  students  that  software 
would  be  used  to  analyze  their  answers  to  detect  cheating  (Bellezza  and  Bellezza,  1989). 
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Possibly,  the  use  of  a  new  technology  is  unsettling  and  the  technology  providing  this 

capability  to  detect  cheating  is  too  new  and  unproven  to  establish  academic  confidence. 

Institutions’  inertia  may  avoid  assessment  methods  they  perceive  will  detract  from  their 

academic  reputations.  Consequently,  proposition  1 : 

PI:  Security  issues  create  a  level  of  comfort  with  the  existing  assessments  and  will 
reinforce  the  institution’s  current  form  of  assessments. 

Proposition  2 

Institutions  are  concerned  with  the  many  administrative  issues  related  to  an  assessment. 
The  issues  can  apply  to  an  electronic  or  a  paper-based  assessment  and  include  items  such  as  the 
faculty’s  time  and  efforts  to  create,  maintain,  deliver,  and  score  an  assessment. 

Faculty  equate  computerized  testing  with  training  or  practice  exercises  for  students  and 
place  machine  or  automated  scoring  of  answer  sheets  in  the  category  of  computerized  testing 
(Stager  and  Mueller,  1991).  Unfortunately,  faculty,  as  a  group,  still  do  not  accept  the  benefit  of 
true  computerized  testing  but  tend  to  be  critical  of  it  (Bicanich  and  others,  1997).  Therefore, 
proposition  2: 

P2:  Faculty  members  and  the  institution’s  leadership  perceive  that  the  administrative 
requirements  of  computer-based  assessments  are  greater  than  those  of  current 
assessments  and  this  will  reinforce  the  institution’s  current  form  of  assessments. 

Proposition  3 

Technology  issues  are  a  major  concern  in  educational  institutions  today.  An  institution 
must  determine  what  types  of  technology  are  appropriate,  now  and  in  the  future,  based  on  its 
mission  and  culture  (Garcia  and  Ratcliff,  1997).  To  accomplish  this,  the  technology  must 
complement  the  institution’s  goals,  mission,  and  its  student  population.  Today,  the  Internet  and 
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web-capable  sites  can  deliver  course  materials,  e-mail,  homework  assignments,  course 
instruction,  student-to-student  and  student-to-instructor  interaction,  and  assessments  (Garcia  and 
Ratcliff,  1997;  Bicanich  and  others,  1997).  However,  few  institutions  conducting  courses  on  the 
Internet  have  ventured  into  conducting  assessments  using  the  same  technology  (Burgess,  1994; 
Spille  and  others,  1997). 

Students  entering  universities  have  a  higher  level  of  computer  literacy  and  technical 
knowledge  than  previous  students  (Komives  and  Peterson,  1997).  Frequently,  the  students  have 
a  better  understanding  of  computers  and  more  knowledge  of  computing  technology  than  the 
average  faculty  member  (Engstrom,  1997).  Consequently,  students  actively  pursuing  an 
academic  education  could  have  sufficient  technical  knowledge  to  actively  engage  in  electronic 
courses.  A  study  conducted  in  1996  indicated  only  38%  of  faculty  used  e-mail  in  their  courses. 
In  addition,  the  faculty  indicated  they  believed  only  31%  of  their  students  knew  how  to  use  the 
World  Wide  Web  (Engstrom,  1997).  This  data  reflects  a  disconnect  between  students’  actual 
technology  capabilities  and  the  faculty's  perceptions  about  the  students'  abilities. 

To  help  satisfy  technology  requirements  on  campus,  institutions  have  been  moving  on 
ambitious  plans  to  have  campus  infrastructures  fully  capable  of  high-speed  computing. 
Unfortunately,  technical  knowledge  and  abilities  of  the  faculty  have  seriously  lagged  behind  the 
technological  advances  (Komives  and  Peterson,  1997;  Bicanich  and  others,  1997). 

Because  institutions  must  evaluate  what  methods  of  delivery  provide  the  student  with  the 
best  opportunity  of  correctly  demonstrating  mastery  of  the  course  material,  innovative 
assessments  should  be  considered  (CUSE,  1997;  Hansen,  1994).  However,  frequently  traditional 
assessment  methods  are  used.  Consequently,  proposition  3: 
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P3:  New  technology  provides  an  unknown  and  unproven  capability  to  the  institution, 
but  as  the  technology  becomes  more  prevalent,  an  institution  may  alter  the  types  of 
assessments  such  that  newer  technology  will  be  utilized. 

Proposition  4 

Cost  issues  are  a  significant  concern  to  universities,  particularly  to  distance  education 
programs.  Costs  of  creating,  maintaining,  and  scoring  the  assessment  are  increasing.  These 
costs  include  the  faculty’s  time  and  effort  to  develop  the  assessment.  Consequently,  an  institution 
should  consider  the  costs  associated  with  a  particular  assessment  and  its  respective  delivery 
method  (Hansen,  1994). 

One  cost  that  affects  institutions  is  the  delivery  cost  for  the  selected  assessment  (Farmer, 
1997).  For  example,  a  paper-based  assessment  delivered  during  a  class  period  by  the  faculty  has 
virtually  no  delivery  costs.  In  distance  education  courses,  delivering  a  paper-based  assessment 
via  the  mail  system  has  preparation,  postage,  and  reception  costs.  An  electronic  assessment 
delivered  via  the  Internet  could  have  much  higher  costs  associated  with  the  delivery  method. 
However,  after  the  initial  installation  of  the  communication  and  computing  capabilities,  the 
delivery  cost  of  each  subsequent  assessment  is  reduced  to  virtually  zero. 

The  computing  hardware  and  software  to  house,  deliver,  score,  and  store  results  for  the 
assessment  incur  costs.  If  this  hardware  and  software  are  purchased  specifically  for  the 
assessment  effort,  then  the  costs  are  higher.  However,  today's  institutions  have  extensive 
computing  capabilities  and  most  likely  will  simply  purchase  the  software  required  to  properly 
process  the  assessment,  thus  lowering  the  direct  costs  attributed  to  a  specific  assessment 
(Bicanich  and  others,  1 997). 
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Actual  assessment  costs  add  a  further  dimension  when  considering  the  various  forms  of 

the  assessment.  Costs  for  the  assessment  also  reflect  a  range  depending  on  the  selected  type  of 

assessment.  Some  assessments  require  more  time  to  score  than  do  others.  One  comparison  is 

between  the  multiple-choice  and  essay  assessments.  In  a  study  conducted  among  Pennsylvania 

high  schools,  software  required  to  create,  disseminate,  score,  and  record  results  had  a  break-even 

point  at  375  students  (Bicanich  and  others,  1997).  Therefore,  proposition  4: 

P4:  Current  assessments  have  known  costs  and  are  likely  to  be  favored  unless  costs  of 
an  alternate  assessment  are  sufficiently  lower  to  alter  the  affordability  and  desirability  of 
the  current  assessment. 

Proposition  5 

Inertia  can  manifest  itself  as  the  tendency  for  an  institution  to  resist  pedagogy  changes 
and  maintain  the  status  quo.  Some  education  professionals  firmly  believe  electronic  testing  is 
inferior  to  paper-based  testing.  These  educators  believe  electronic  testing  is  expensive,  produces 
lower  student  achievement,  is  limited  to  assessing  information  recall  and  recognition,  separates 
the  student  evaluation  process  from  instruction,  increases  student  anxiety  about  the  testing 
process,  and  removes  the  teacher  from  the  evaluation  process.  Educational  research  has  cast 
doubt  on  all  these  criticisms,  but  educators  still  persist  in  avoiding  computer-based  testing 
(Mojkowski,  1990;  White,  1990;  Bicanich  and  others,  1997;  Stager  and  Mueller,  1991;  Hansen, 
1994;  Kean,  1994). 

Computers  offer  new  and  potentially  valuable  methods  of  delivering  the  assessment  to 
students.  This  area  offers  institutions  the  opportunity  to  address  the  underdeveloped  pedagogy  of 
electronic  examinations  (Wubbels  and  Girgus,  1997).  Additionally,  computers  can  provide 
interactive  assessments  and  can  handle  a  far  greater  number  of  students.  Whether  using  a 
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multiple-choice,  essay,  or  other  forms  of  assessments,  new  technologies  can  assist  the  faculty  in 

performing  the  assessment  evaluation  of  students  (CUSE,  1997). 

In  addition,  institution  politics  can  be  such  that  research  outcomes  advocating  a  change  in 

policy  or  program  will  be  criticized  based  on  research  methodology  as  opposed  to  the  findings 

themselves  (Jacobi  and  others,  1987).  Consequently,  proposition  5: 

P5:  Inertia  in  an  institution  will  reinforce  the  current  assessments  used  within  the 
institution  and  will  not  alter  from  this  course  unless  technology  or  cost  issues  provide 
sufficient  motivation  to  do  so. 

Summary 

As  the  literature  review  demonstrated,  institutional  assessments  can  be  difficult  to 
change.  For  the  development  of  the  Assessment  Selection  Model  (ASM),  inertia  is  used  to 
demonstrate  the  tendency  of  institutions  to  resist  change.  Change  is  seen  by  some  as  an 
implication  that  something  is  wrong  or  deficient  in  the  current  methodology.  In  the  area  of 
assessments,  it  is  possible  that  technology  has  produced  the  ability  to  enhance  the  assessment 
process  as  opposed  to  correcting  a  deficiency.  The  remaining  four  constructs  in  the  model  are 
cost  issues,  administrative  issues,  security  issues,  and  technology  issues. 

These  four  constructs  apply  pressure  on  the  institution’s  inertia.  The  more  that  high 
assessment  costs  become  an  issue,  the  more  likely  the  institution  will  be  to  change  assessments. 
For  example,  in  distance  education,  it  may  be  less  expensive  to  e-mail  an  assessment  to  a  student 
as  opposed  to  mailing  a  paper  examination. 

Administrative  issues  include  the  tasks  associated  with  creating,  maintaining,  scoring, 
and  storing  the  results  in  the  database.  For  courses  with  many  students,  the  administrative  costs 
per  assessment  can  go  down  using  advanced  computing  technologies.  The  more  courses  that 
utilize  this  technology  the  less  cost  is  directly  traced  and  amortized  to  a  specific  assessment. 
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Security  issues  are  of  paramount  concern  for  institutions.  Academic  standards  and 
reputation  depend  on  quality  assessments  executed  in  a  professional  and  secure  manner  (Kean, 
1994).  Electronic  assessments  can  also  provide  a  high  level  of  security  protection  for  the 
institution.  But,  for  an  institution  to  break  free  of  the  inertia,  the  proof  of  this  protection  will 
need  to  be  significant. 

Finally,  new  technologies  enable  electronic  assessments  to  be  practical.  As  these 
technologies  become  more  prevalent  and  well  known,  institutions  may  feel  more  comfortable 
with  them.  Then,  and  probably  only  then,  will  some  institutions  feel  secure  in  performing 
electronic  assessments. 

The  next  chapter  of  the  thesis  details  the  methodology  for  validating  the  model  and  the 
plan  for  gathering  the  data.  The  chapter  also  details  how  the  target  population  was  selected  and 
references  the  interview  questions  asked  to  the  respondents. 
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III.  Methodology 


Introduction 

This  chapter  describes  the  overall  methodology  used  to  conduct  the  research.  It  describes 
the  process  of  identifying  and  selecting  the  target  sample  population,  conducting  the  initial 
validation  of  the  Assessment  Selection  Model  (ASM),  and  testing  the  model. 

Research  Design  and  Implementation 
Overview 

This  is  an  exploratory  study  primarily  because  a  review  of  the  literature  has  not  disclosed 
previous  research  in  this  area.  Consequently,  the  current  research  uses  existing  research  findings 
to  develop  a  potentially  useful  model  for  describing  the  behavior  of  academic  institutions 
concerning  assessment  selection  decisions.  Five  specific  propositions  were  then  developed 
based  on  this  model.  The  study  then  collected  descriptive  data  to  evaluate  the  propositions.  The 
results  of  these  propositions  were  used,  in  turn,  to  evaluate  the  Assessment  Selection  Model. 

Interview  questions,  located  at  Appendix  A,  were  developed  to  query  the  selected 
institutions  as  to  possible  issues  effecting  their  choice  of  student  assessments.  A  telephone 
interviewing  technique  was  employed  to  provide  the  data.  This  technique  produces  rich  data  that 
is  appropriate  for  this  type  of  study.  The  targeted  subjects  of  the  instrument  were  selected  from  a 
list  of  accredited  academic  institutions  that  conduct  distance  education  courses  (Burgess,  1994; 
Spille  and  others,  1 997). 
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Characteristics  of  Qualitative  Research 


Qualitative  research  is  a  form  of  research  design  to  make  sense  out  of  some  phenomenon, 
while  disrupting  the  environment  as  little  as  possible.  Qualitative  research  uses  inductive 
reasoning  to  provide  plausible  explanations  of  the  events.  The  following  paragraphs  list  five 
characteristics  of  qualitative  research  (Merriam,  1998;  Schmitt  and  Klimoski,  1991). 

The  first  characteristic  of  qualitative  research  is  its  concern  for  individuals  and  their 
interactions  with  their  environment.  The  qualitative  researcher  is  interested  in  the  ways 
individuals  make  sense  of  their  environment.  This  study  is  interested  in  revealing  the  logical 
decision  making  processes  used  to  select  assessments. 

The  second  characteristic  of  qualitative  research  is  an  interactive  method  of  collecting 
data  and  conducting  the  data  analysis.  The  interaction  with  the  subjects  of  the  research  can  be 
more  personal  and  dynamic  than  with  computers  or  questionnaires. 

The  third  characteristic  is  that  qualitative  research  normally  requires  that  the  researcher 
and  subject  be  physically  co-located.  In  this  study,  co-location  is  not  possible.  To  alleviate  this 
requirement,  a  telephone  interview  approach  is  used. 

The  fourth  characteristic  is  that  qualitative  research  primarily  focuses  on  building 
concepts  or  theories  rather  than  testing  them.  That  is  also  true  of  this  study.  Since  there  are 
currently  no  models  addressing  this  issue,  this  research  builds  and  explores  such  a  model. 

Finally,  the  qualitative  study  is  based  on  rich  data  derived  from  the  sample  population  as 
compared  to  the  hard  numbers  generated  by  the  quantitative  researcher.  Each  of  the 
characteristics  of  the  qualitative  research  is  indicative  of  the  present  study.  Therefore,  a 
qualitative  study  is  the  appropriate  form  of  research  to  answer  the  research  questions. 
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Major  Types  of  Qualitative  Research 


There  are  essentially  five  basic  types  of  research  in  the  area  of  qualitative  research  in 
education.  Table  6  provides  the  five  types  and  their  respective  characteristics: 

Table  6.  Types  of  Qualitative  Research 


Type 

Characteristics 

Basic  or  generic 

-  Includes  description,  interpretation,  and  understanding 

-  Identifies  recurrent  patterns  in  the  form  of  themes  or  categories 

-  May  delineate  a  process 

Ethnography 

-  Focuses  on  society  and  culture 

-  Uncovers  and  describes  beliefs,  values,  and  attitudes  that  structure 
behavior  of  a  group 

Phenomenology 

-  Is  concerned  with  essence  or  basic  structure  of  a  phenomenon 

-  Uses  data  that  are  the  participant's  and  the  investigator's  firsthand 
experience  of  the  phenomenon 

Grounded  Theory 

-  Is  designed  to  inductively  build  a  substantive  theory  regarding 
some  aspect  of  practice 

-  Is  "grounded"  in  the  real  world 

Case  Study 

-  Is  intensive,  holistic  description  and  analysis  of  a  single  unit  or 
bounded  system 

-  Can  be  combined  with  any  other  of  the  above  types 

(Merriam,  1998:12) 


The  basic  or  generic  qualitative  study  is  representative  of  the  current  study.  This  form 
tries  to  uncover  the  process  or  understand  the  phenomenon  being  studied.  The  study  draws 
information  from  other  research  to  build  a  foundation  of  knowledge  about  the  subject.  The 
researcher  using  interviews  or  observations  collects  additional  data.  Then  the  data  is  analyzed 
for  patterns. 
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Construct  Validation 


An  initial  assessment  selection  model  was  developed  from  the  literature.  Through 
meetings  with  the  author’s  thesis  committee,  modifications  were  made  to  more  clearly  represent 
the  constructs  involved. 

Two  groups  of  three  graduate  students  validated  the  revised  model  constructs.  The  first 
group,  validated  the  revised  list  of  constructs  and  issues.  Each  student  was  given  the  constructs 
and  a  description  of  each  construct.  In  addition,  the  student  was  given  the  individual  issues  that 
tied  to  the  constructs.  The  students  mapped  the  specific  issues  to  constructs  based  on  the 
definitions  provided  with  the  constructs.  The  individual  issues  were  the  responses  institutional 
representatives  are  expected  to  use  as  the  issues  affecting  their  decision  making  process.  The 
aggregated  issues  were  categorized  into  the  constructs.  As  a  result  of  this  validation  effort, 
changes  were  made  to  the  model. 

The  group  indicated  several  ambiguities  in  construct  descriptions  and  several  overlapping 
issues  initiating  unnecessary  confusion.  Consequently,  nine  issues  were  moved  to  other 
constructs  that  provided  more  logical  mapping.  Also,  six  issues  were  deleted  due  to  overlapping 
coverage.  Two  issues  were  added  to  provide  clarity  in  exactly  what  the  issues  was  concerned 
with  and  nine  issues  were  renamed  to  provide  a  more  descriptive  and  clearer  concept  of  the 
issue. 

The  second  group  was  also  given  the  construct  and  the  individual  issues  from  the  newly 
revised  model.  This  iteration  found  a  logical  and  agreeable  mapping  between  the  constructs  and 
issues.  The  complete  issues  and  constructs  used  for  the  model  are  located  in  Table  1  in  Chapter 
II.  The  completed  model  is  located  in  Figure  1  in  Chapter  II. 
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Next,  interview  questions  were  created  to  extract  the  assessment  selection  behavior  of  the 
chosen  institutions.  A  pre-test  of  the  interview  questions  was  completed  with  a  local  college 
representative. 

Sample  Population 

This  study’s  target  population  was  derived  from  listings  in  two  books.  The  first  book, 
External  Degrees  in  the  Information  Age,  lists  140  institutions  conducting  distance  education 
courses.  The  second  book,  The  Oryx  Guide  to  Distance  Learning:  A  Comprehensive  Listing  of 
Electronic  and  Other  Media- Assisted  Courses  lists  298  institutions  offering  some  form  of 
distance  education. 

The  delivery  methods  of  the  438  institutions  included  audiocassette,  audio  conferencing, 
audiographic  conferencing,  broadcast  television,  computer  conferencing,  electronic  mail, 
interactive  audio/video,  online  services,  radio  broadcast,  satellite  network,  telephone  contact,  or 
videocassette. 

Described  below  are  the  reduction  criteria  used  to  select  the  institutions  comprising  the 
study’s  sample  population.  First,  the  institution  must  have  utilized  some  form  of  electronic 
course  format.  Many  of  the  institutions  listed  in  the  books  offered  only  videocassette  or 
television  courses.  These  types  of  institutions  were  omitted  from  the  sample  because  they  are 
not  a  typical  representation  of  the  population  of  interest.  Only  schools  offering  at  least  one 
electronic  course  would  likely  provide  insight  into  the  research  questions. 

Second,  all  of  the  institutions  selected  for  the  sample  population  must  be  four-year 
institutions  accredited  by  a  reputable  academic  accreditation  organization.  Because  a  goal  of  the 
study  is  to  provide  generalizability  to  the  general  academic  population,  the  sample  population 
must  reflect  the  population  of  interest. 
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Finally,  the  selected  institutions’  representatives  must  have  shown  the  willingness  to 
participate  in  the  research.  The  issue  of  participation  can  cause  slight  concern  for  reliability 
since  the  institutions’  representatives  that  responded  might  not  provide  the  ability  to  generalize. 
However,  for  an  exploratory  study,  this  was  deemed  to  be  an  acceptable  limitation.  The  chosen 
institutions  for  the  sample  population  are  listed  at  Appendix  B.  The  interview  sample  was 
selected  using  a  random  number  cycling  through  the  sample  list,  at  Appendix  B,  until  20 
participating  universities  were  interviewed.  The  identities  of  the  participating  universities  were 
intentionally  excluded  to  prevent  a  reader  from  correlating  specific  comments  to  a  particular 
university. 


Interviewing 

Frequently,  interviews  are  required  when  behavior  is  not  available  to  be  observed.  This 
could  occur  when  studying  past  events  or  when  it  is  not  possible  for  the  researcher  to  observe  the 
event.  Table  7  demonstrates  a  broad  range  of  interview  structures  (Merriam,  1998). 

Table  7.  Continuum  of  Interview  Structures 


HighlyStructured/Standardized 


Semistructured 


Unstructured/Informal 


► 


Wording  of  questions 
Predetermined 

Mix  of  more-and  less- 
structured  questions 

Open-ended  questions 

Order  of  questions  predetermined 

Flexible,  exploratory 

Oral  form  of  a  survey 

More  like  a  conversation 

(Merriam,  1998) 


As  the  continuum  demonstrates,  the  unstructured  interview  is  more  directed  to 
exploratory  research.  This  method  is  useful  when  little  is  known  about  the  phenomenon  under 
study.  The  types  of  questions  providing  the  best  relationship  between  the  study  and  domain 
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knowledge  are  open-ended  questions.  This  type  expands  the  knowledge  base  about  the  subject 
without  hampering  the  information  gathering  based  on  structured,  but  erroneous  questions. 

There  are  several  advantages  and  disadvantages  in  conducting  interviews.  Table  8 
contains  a  subset  of  the  advantages  published  by  Hughes.  Interviews: 


Table  8.  Advantages  of  Interviews 


Obtain  large  amounts  of  expansive  and  contextual  data  quickly 

Facilitate  cooperation  from  research  subject _ 

Facilitate  access  for  immediate  follow-up  data  collection  for 

clarification  and  omissions  _ 

Are  useful  for  discovering  complex  interconnections  in  social 
relationships 

Facilitate  analysis,  validity  checks,  and  triangulation _ 

Facilitate  discovery  of  nuances  in  culture _ 

Offer  great  utility  for  uncovering  the  subjective  side _ 

(Hughes,  1996) 


The  advantages  listed  in  the  table  adequately  demonstrate  the  benefits  using  the 
interviewing  process  in  the  current  study.  Because  this  study  is  exploratory  and  discovers 
aspects  of  a  phenomenon,  interviewing  does  have  the  advantage  of  extracting  relevant  data  from 
the  selected  sample  population.  Telephone  interviewing  is  also  the  most  cost  effective  method  of 
interviewing  subjects  in  a  large  geographical  area  (Judd,  1991). 

Table  9  contains  the  disadvantages  as  published  by  Hughes. 

Table  9.  Disadvantages  of  Interviews 


Depend  on  the  cooperation  of  a  small  group  of  key  informants _ 

Difficult  to  replicate 
Procedures  are  not  always  explicit 

Data  often  subject  to  observer  effects;  obtrusive  and  reactive _ 

Dependent  on  the  ability  of  the  researcher  to  be  resourceful, 

systematic,  and  honest;  to  control  bias _ 

(Hughes,  1996) 
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The  limited  disadvantages  are  more  than  offset  by  the  utility  and  benefit  provided  by  the 
process.  The  only  concern  is  that  a  researcher  may  unintentionally  affect  subject  responses 
causing  suspect  data. 

Recording  data  is  important  to  the  interviewing  process.  There  are  three  ways  to  record 
the  interview:  use  a  tape  recorder,  take  notes  during  the  interview,  write  notes  after  the  interview 
(Merriam,  1998;  Hughes,  1996).  Since  this  study  uses  a  telephone  interview  process  and 
produces  a  tremendous  amount  of  data,  the  third  method  is  ruled  out.  Tape  recording  is  the  most 
effective  in  recording  the  subject's  actual  comments  and  can  be  replayed  to  ensure  accuracy. 

This  method,  however,  does  have  the  tendency  of  initially  making  the  subject  nervous. 
Consequently,  the  interviewer  took  notes  during  the  interview  and  typed  the  rest  immediately 
following  the  interview  to  ensure  completeness. 

Interview  Questions 

The  interview  consists  of  a  brief  introduction  of  the  study  and  the  interviewer.  This  is 
followed  by  several  demographic  or  general  questions  to  properly  frame  the  institutions  in  the 
study.  Then  25  questions  related  to  the  specific  constructs  of  the  study  are  addressed.  Each 
question  was  written  to  provide  information  to  evaluate  the  five  stated  propositions.  The 
questions  are  at  Appendix  A. 

Summary 

A  qualitative  research  method  based  on  an  interactive,  telephone  conversation  between 
the  university  representative  and  the  researcher  was  used.  To  accomplish  this,  25  questions  were 
developed  and  validated  to  elicit  the  necessary  information  from  the  universities.  The  researcher 
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selected  55  universities  to  represent  the  larger  population  of  accredited,  four-year  universities. 
The  universities,  listed  at  Appendix  B,  were  selected  because  of  their  ability  to  provide  the  rich 
data  required  in  the  study.  Of  the  fifty-five  universities,  twenty  were  randomly  selected  to 
participate  in  the  study.  The  results  of  the  interview  are  present  in  the  next  chapter. 
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IV.  Findings  and  Analysis 


Introduction 

Fifty-five  universities  were  selected  to  represent  the  greater  population  of  institutions 
offering  course  credit  for  distance  education  courses.  Of  these,  twenty  universities  were 
interviewed  during  the  research.  The  university  representative’s  comments  were  used  to  develop 
profiles  to  evaluate  the  research  propositions  and  validate  the  ASM. 

The  universities  included  both  public  and  private  universities.  The  in-state  annual  (1997- 
1998)  tuition  for  the  interviewed  universities  ranged  from  about  $2,000  to  more  than  $20,000. 
The  undergraduate  student  population  of  the  universities  ranged  from  about  4,500  to  over 
35,000.  In  addition,  the  schools  ranged  in  rankings  from  several  in  the  top  50  in  the  country  to 
schools  in  the  fourth  tier  (U.S.  News  Online,  1997).  For  this  study,  small  schools  are  those  that 
have  10,000  or  less  students.  Medium  schools  have  more  than  10,000  but  less  than  or  equal  to 
20,000.  Finally,  large  schools  have  more  than  20,000  undergraduate  students. 

Based  on  the  results,  the  interviewing  institutions  were  grouped  into  two  categories.  One 
represented  those  institutions  that  were  slow  to  adopt  nontraditional  assessment  methods.  The 
second  group  consisted  of  those  institutions  that  more  rapidly  adopted  newer  assessment 
methods.  The  differentiation  between  the  groups  is  based  on  the  technology  adoption  tendencies 
expressed  by  the  representatives  of  the  schools. 
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University  Profiles 


Slower  to  Adopt 

The  characteristics  of  the  group  that  tended  not  to  adopt  the  more  innovative  or  non- 
traditional  assessments  are  shown  in  Table  10. 

Table  10.  Slow  to  Adopt 


Slow  to  Adopt 

Region 

Size 

Ranking  (Tier) 

South 

Large 

2 

North 

Large 

2 

South 

Small 

3 

South 

Medium 

3 

South 

Medium 

3 

South 

Medium 

3 

North 

Small 

4 

East 

Medium 

4 

South 

Large 

4 

In  general,  the  group  of  universities  listed  in  Table  10  are  concerned  with  security  issues 
as  the  primary  consideration  with  regards  to  assessments  in  distance  education.  As  a  group,  they 
tend  to  be  risk  averse.  In  five  of  the  schools,  security  issues  appeared  to  be  paramount  in 
determining  their  decision  not  to  pursue  new  possibilities  in  distance  education.  The 
representatives  from  these  schools  used  descriptions  such  as:  “they  do  not  know  if  the  student’s 
cheating”;  “this  is  a  show-stopper,  too  many  dangers”;  “put  the  distance  education  students 
through  the  ringer”;  “they  do  not  discuss  new  assessments  because  they  cannot  get  that  far 
because  of  security”;  “they  are  going  to  wait  until  they  get  down  the  road”;  “they  could  only  do 
electronic  testing  if  it  were  proctored”;  and  “no  infallible  way  to  detect  or  prevent  cheating.” 

A  second  notable  area  of  consideration  was  the  idea  of  inertia.  Representatives  from  six 
of  the  nine  schools  mentioned  this  as  preventing  progress  in  the  use  of  technology  in  the 
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assessment  process.  In  one  of  the  six,  the  sheer  size  of  the  university  was  noted  as  causing 
stagnation  of  progress.  In  another  of  the  six,  the  faculty  had  to  be  frequently  reassured  they  were 
in  control  of  the  class  and  that  technology  would  not  be  used  or  allowed  to  encroach  upon  that 
domain.  Still  further,  representatives  from  the  three  remaining  universities  mentioned  this 
phenomenon  as  slowing  or  retarding  the  adoption  of  new  technology. 

Technology  concerns  constituted  a  third  important  issue  as  viewed  by  the  sample  as  a 
whole.  Representatives  from  five  of  the  nine  schools  indicated  that  the  schools  were  limited  in 
technology  capabilities  and  faculty  abilities.  Representatives  from  the  four  remaining  schools 
indicated  that  they  had  sufficient  technology  to  complete  the  job  but  did  not  indicate  the  ability 
level  of  the  faculty.  The  technological  capabilities  and  abilities  of  this  group  of  schools  were 
varied  but  tended  to  be  limited.  These  limitations  included  limited  availability  of  the  technology 
to  faculty  and  limited  infusion  of  technology  into  course  curriculum.  The  limited  abilities  are  in 
terms  of  the  faculty’s  computing  knowledge  and  expertise.  This  group’s  courses,  while  making 
some  use  of  technology,  remained  more  technologically  limited  than  the  second  group's  courses. 
The  first  group  also  tended  to  focus  more  on  the  mechanics  of  technology  than  on  the 
pedagogical  implications  of  the  technology.  The  technology  limitation  issue  was  also  more 
prevalent  in  southern  schools  than  in  schools  in  the  other  regions  of  the  country. 

Administrative  issues  were  the  fourth  most  paramount  issue.  Compared  to  the  second 
group,  faculties  were  not  encouraged  to  utilize  technology  to  enhance  the  learning  experience. 
Faculty  training  and  access  to  computing  resources  were  also  significant  concerns  to  all  of  these 
schools.  Representatives  of  two  schools  specifically  mentioned  the  need  to  build  and  maintain  a 
support  staff  for  the  faculty.  Representatives  in  five  of  the  nine  schools  discussed  the  increase  in 
faculty  workload  from  distance  education  courses  as  contributing  to  the  university’s  problems. 
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Cost  issues  were  presented  as  the  least  significant  issue  in  this  group.  However,  costs 
tended  to  be  much  more  significant  for  this  group  than  to  the  second  group.  Representatives 
from  three  of  the  schools  thought  financial  matters  were  extremely  important  and  two  thought 
that  costs  were  not  important.  The  remaining  four  were  somewhat  concerned. 


Quicker  to  Adopt 

Table  1 1  lists  the  schools  who  are  quicker  to  adopt  the  more  innovative  or  non-traditional 
assessments. 


Table  11.  Quicker  to  Adopt 


Quicker  to  Adopt 

Region 

Size 

Ranking  (Tier) 

Northeast 

Medium 

1 

Mid-west 

Large 

2 

East 

Large 

2 

Mid-west 

Large 

2 

Mid-west 

Large 

2 

East 

Large 

2 

East 

Small 

3 

Mid-west 

Medium 

3 

West 

Medium 

3 

Northeast 

Small 

4 

West 

Small 

4 

The  quicker  to  adopt  group  is  also  primarily  concerned  with  security  issues  but  not  to  the 
degree  of  the  first  group.  They  tended  to  discuss  the  issue  as  requiring  resolution  or  some 
consideration  rather  than  as  an  insurmountable  obstacle.  How  each  school  addressed  the  security 
issue  was  quite  varied.  Many  school  faculties  mitigated  the  concern  over  who  the  test  taker  is  by 
building  the  assessment  process  from  the  beginning  with  distance  education  in  mind.  For 
example,  instructors  greatly  reduced  the  use  of  midterms  and  finals.  Projects,  electronic 
discussions,  and  context-sensitive  experiences  were  used  as  evaluation  tools.  Also,  randomly 
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generated,  equivalent  exams  were  used  to  prevent  two  students  from  receiving  the  same  exam. 
One  school’s  representative  indicated  a  student  might  be  able  to  find  someone  to  cheat  for  a 
student  in  one  course  but  not  for  an  entire  degree.  Another  school’s  representative  mentioned 
that  in  an  electronic  setting,  a  student  would  need  to  receive  illegal  assistance  from  day  one  or 
else  the  instructor  could  see  the  disparity  in  writing  and  intellectual  styles  during  the  term. 

Administrative  issues  were  the  second  most  substantial  concern  for  this  group.  In  this 
category,  the  faculties  were  encouraged  to  utilize  technology  to  enhance  the  learning  experience 
more  so  than  the  schools  in  the  first  group.  Faculty  training  and  adequate  support  staff  were  of 
significant  concern  to  all  of  the  schools  in  the  group.  However,  four  school  representatives 
specifically  mentioned  the  support  staff.  A  support  staff  is  needed  to  provide  the  level  of 
services  required  by  the  faculty  to  create  the  product  desired.  In  some  situations,  it  is  the  staff 
that  directly  allows  a  quality  product  to  be  created.  Faculty  at  five  schools  in  this  group  spent 
time  and  money  determining  how  best  to  demonstrate  a  student's  mastery  of  the  material.  Just  as 
in  the  two  schools  in  the  first  group,  this  evaluation  process  is  an  important  issue  that  can 
produce  pedagogical  changes  in  the  future. 

Technology  concerns  were  the  third  most  important  issue  as  viewed  by  this  group  of 
universities.  The  technological  capabilities  of  this  group  are  more  substantial  than  the  first 
group.  Representatives  from  eight  of  the  eleven  schools  indicated  the  infrastructure  is  in  place 
and  provides  the  required  level  of  capability.  Although  representatives  from  six  of  the  eleven 
schools  specifically  mentioned  student  capabilities,  there  were  fewer  self-imposed  limitations 
due  to  the  student's  computing  capability  as  compared  to  the  first  group. 

The  fourth  area  of  significant  consideration  was  the  issue  of  inertia.  Similar  to  the  first 
group,  representatives  from  three  schools  mentioned  inertia  as  slowing  or  retarding  the  adoption 
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of  new  technology.  However,  representatives  from  seven  of  the  remaining  schools  in  the  group 
indicated  that  they  would  not  and  do  not  resist  pedagogical  changes  associated  with  the  infusion 
of  technology  into  the  curriculum.  The  representative  from  the  eleventh  school  did  not  mention 
this  issue.  This  group  tended  to  be  more  open  to  the  technology  available  and  more  willing  to 
employ  the  technology.  The  faculty  in  this  group  tended  not  to  be  as  resistant  to  changes  in 
delivery  methods.  Of  course,  this  tendency  was  more  pronounced  in  some  schools  than  in 
others. 

Cost  issues  appear  to  be  the  least  significant  issue  for  this  group.  Representatives  from 
only  two  schools  thought  financial  matters  were  extremely  important.  Representatives  from 
three  schools  believed  that  cost  played  a  minor  role.  Representatives  from  the  six  remaining 
schools  indicated  that  cost  was  not  significantly  important.  This  group  also  tended  to  focus  on 
the  opportunity  costs  of  the  new  technology  and  take  a  long-term  perspective.  Specifically,  costs 
were  viewed  as  decreasing  in  the  long  run  as  overhead  and  indirect  cost  pools  were  reduced. 

Table  12  summarizes  the  universities  and  the  chosen  order  of  importance  of  assessment 
selection  issues. 
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Table  12.  Summary  of  Schools  and  Issue  Order 


|  Slow  to  Adopt 

Quicker  to  Adopt  j 

Region 

Size 

Ranking  (Tier) 

Region 

Size 

Ranking  (Tier) 

South 

Large 

2 

Northeast 

Medium 

1 

North 

Large 

2 

Mid-west 

Large 

2 

South 

Small 

3 

East 

Large 

2 

South 

Medium 

3 

Mid-west 

Large 

2 

South 

Medium 

3 

Mid-west 

Large 

2 

South 

Medium 

3 

East 

Large 

2 

North 

Small 

4 

East 

Small 

3 

East 

Medium 

4 

Mid-west 

Medium 

3 

South 

Large 

4 

West 

Medium 

3 

Northeast 

Small 

4 

West 

Small 

4 

Issue  Order 

Issue  Ore 

ler 

Security 

Security 

Inertia 

Administrative 

Technology 

Technology 

Administrative 

Inertia 

Costs 

Costs 

Propositions  Explored 
Proposition  1 

PI :  Security  issues  create  a  level  of  comfort  with  the  existing  assessments  and  will 

reinforce  the  institution’s  current  form  of  assessments. 

The  proposition  was  supported  by  the  data.  Representatives  from  all  universities 
mentioned  security  issues  as  points  of  concern  and  virtually  all  touched  on  the  potentially 
limiting  factor  that  security  concerns  cause.  Security  concerns  ranged  from  being  a  barrier  to  not 
really  being  a  concern  but  something  requiring  attention. 

At  one  end  of  the  spectrum,  security  is  such  a  sensitive  issue  that  faculties  felt  that  they 
cannot  risk  security  breaches  by  trying  new  technologies.  For  example,  a  representative  from  a 
small  eastern  school  mentioned  that  security  was  so  significant  a  concern  that  local  students  were 
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required  to  come  on  campus  to  complete  some  assignments.  The  representative  from  the  school 
admitted  that  this  is  certainly  not  what  distance  education  is  fully  capable  of  --  complete 
independence  of  the  campus.  A  representative  from  a  small  southern  school  indicated  that  there 
are  simply  too  many  dangers  involved  with  electronic  assessments. 

At  the  other  end  of  the  spectrum,  representatives  from  several  schools  felt  that  they  have 
addressed  the  issue  satisfactorily  and  the  security  provided  in  the  distance  education  arena  is 
equivalent  to  that  in  the  classroom.  Representatives  from  two  mid-western  universities,  one 
large  and  one  medium  sized,  stated  that  although  they  do  not  know  with  certainty  who  is  at  the 
other  end  of  the  keyboard,  they  also  do  not  know  if  all  300  students  really  belong  in  the  lecture 
hall  taking  an  exam.  This  suggests  a  possible  double  standard.  Are  distance  education  courses 
and  faculty  held  to  a  higher  security  standard  than  their  on-campus  counterparts? 

Representatives  from  two  schools  mentioned  that  same  concern.  The  representative  from  one  of 
the  two  schools  said  distance  education  fights  to  maintain  credibility  and  cannot  afford  any 
security  breeches.  The  representative  from  the  second  school  talked  about  its  reputation  being 
judged  when  a  graduate  is  interviewed  for  a  job.  This  representative  from  the  distance  education 
program  felt  that  it  could  not  sustain  its  reputation  if  it  had  a  graduate  that  did  not  know  what  he 
or  she  was  supposed  to  know. 

Between  the  two  ends  of  the  spectrum  are  the  remaining  universities.  The  differentiation 
of  universities  and  their  location  on  the  spectrum  is  more  related  to  specific  security  concerns 
and  how  these  concerns  were  mitigated  and  less  with  the  pursuit  of  perfect  security. 

Representatives  from  five  schools  specifically  mentioned  concerns  for  hackers  or 
students  illegally  entering  and  manipulating  the  computer  systems  or  data  files,  including  exams. 
To  counter  this  issue,  the  instructors  either  limited  the  information  available  or  use  personal 
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logins  and  passwords  to  mitigate  the  risk.  The  five  schools  included  small,  medium,  and  large 
mid-westem  universities,  a  medium  southern  school,  and  a  small  eastern  school. 

Representatives  from  twelve  schools  specifically  indicated  that  they  are  concerned  with 
knowing  who  is  at  the  other  end  on  the  keyboard.  The  ways  that  schools  resolved  the 
identification  issue  were  varied.  The  representative  from  one  medium  sized  northeastern 
university  advanced  the  idea  that  a  professor  could  know  a  student  much  better  in  the  electronic 
environment  than  the  lecture  hall  environment.  Because  an  instructor  reading  and  discussing 
course  material  in  a  chat  room,  virtual  room,  or  using  other  real-time  or  asynchronous 
communication  methods  would  be  more  familiar  with  the  student’s  opinions  and  writing  skill 
than  in  a  more  passive  lecture  environment.  In  addition  to  the  discussions,  each  course  has 
several  projects  or  writing  assignments  for  the  student  to  complete.  During  the  course  of  the 
student's  assignments,  the  instructor  will  receive  indicators  about  the  grammatical  abilities, 
intellectual  level,  problem  solving  process,  and  word  choice  the  student  displays.  Consequently, 
when  an  exam  or  assignment  is  done,  the  instructor  should  be  able  to  detect  significant  variations 
that  would  be  caused  by  someone  other  than  the  student  completing  the  assignment.  In  the 
lecture  hall,  the  student  may  not  say  anything  or  very  little.  Then  when  it  comes  time  for  a  test 
or  a  written  assignment,  the  professor  really  has  no  basis  from  which  to  evaluate  whether  the 
student  is  capable  of  such  work. 

This  insightful  method  of  recognizing  quality  variations  in  a  student’s  work  is  an 
interesting  perspective.  However,  it  places  an  enormous  burden  on  the  professor.  The  professor 
must  be  intimately  involved  with  the  students  and  their  assignments.  Representatives  from 
several  universities  indicated  that  the  faculty  is  already  inundated  with  work  and  might, 
therefore,  not  select  this  method. 
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Representatives  from  eight  schools  raised  another  issue;  how  can  one  be  sure  that  the 
student  did  not  use  additional  materials  for  the  exam?  This  concern  is  alleviated  differently  by 
the  various  schools.  Two  schools,  a  large  eastern  and  medium  mid-western  university,  simply 
require  students  to  go  to  an  approved  computer  lab.  This  method  provides  the  desired  proctoring 
services  that  the  school  feels  will  ensure  that  the  student  acquires  no  advantage.  Two  eastern 
schools,  one  medium  and  one  small,  use  time  limits  for  the  test  duration.  This,  they  feel,  will 
produce  sufficient  assurances  that  the  student  must  come  prepared. 

A  small  eastern  and  a  small  western  school  minimizes  student  cheating  by  asking 
questions  that  require  student's  experiences  to  be  discussed  within  the  context  of  the  course 
material  or  projects.  These  two  universities  also  make  extensive  uses  of  projects,  both  individual 
and  group.  The  faculty  of  the  eastern  school  also  requires  telephone  contact  between  the 
students  and  faculty  during  the  term.  The  telephone  contact  is  design  to  build  a  rapport  that  will 
provide  a  more  personal  relationship.  The  representative  from  the  school  believes  it  will  reduce 
potential  security  problems. 

Representatives  from  five  schools  mentioned  that  the  faculty  is  concerned  with  the  issue 
of  copyrights.  The  faculty  is  concerned  with  producing  electronic  materials  for  courses  and  then 
having  the  material  made  available  to  the  larger  population  with  no  royalty  or  financial 
consideration  for  themselves.  Additionally,  one  of  the  representatives  indicated  that  information 
available  on  the  WWW  almost  certainly  guarantees  the  information  will  be  duplicated  many 
times  over.  One  medium  sized  southern  school  discussed  a  different  perspective  on  this  issue.  Is 
a  faculty  member  primarily  responsible  for  transmitting  subject  matter?  Or  is  a  faculty  member's 
primary  responsibility  to  certify  the  mastery  of  subject  matter?  This  later  perspective  changes 
the  focus  from  the  preoccupation  with  developing  and  delivering  the  course  material  to  more 
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thoroughly  evaluating  mastery  of  the  course  material  by  the  student  indicating  an  interesting  shift 
in  perspective.  A  student  or  non-student  could  benefit  from  reading  the  course  material  on  the 
Internet,  but  could,  in  no  way,  receive  college  credit  from  obtaining  the  material.  For  the  student 
to  receive  the  credit,  the  student  must  enroll  and  demonstrate  mastery. 

One  small  northern  school  was  concerned  with  foreign  cultures  and  their  perception  of 
cheating.  As  Internet  courses  become  more  prevalent  and  people  of  other  cultures  take  more  and 
more  courses  from  American  universities,  schools  should  be  concerned  with  other  cultural 
aspects  and  beliefs  of  security  and  cheating. 

One  large  northern  school  said  that  the  faculty  sometimes  puts  the  distance  education 
students  through  more  stringent  security  measures  than  they  do  for  on  campus  students.  The 
faculty  members  wanted  to  be  very  sure  that  the  student  getting  the  grade  was  the  student  taking 
the  exam.  The  representative  of  the  school  also  stated  that  the  faculty  wanted,  in  essence,  to 
ensure  that  there  was  no  possible  way  that  a  student  could  cheat. 

Proposition  2 

P2:  Faculty  members  and  the  institution’s  leadership  perceive  that  the  administrative 
requirements  of  computer-based  assessments  are  greater  than  those  of  current 
assessments  and  this  will  reinforce  the  institution’s  current  form  of  assessments. 

This  proposition  is  supported.  The  interviews  revealed  two  significant  concerns  with 

respect  to  administrative  issues  that  seemed  to  hinder  the  use  of  different  assessments. 

Representatives  from  eight  universities  specifically  mentioned  the  first,  quality  of  the 

assessment.  Secondly,  the  representatives  from  six  of  the  schools  explicitly  mentioned  the  need 

for  a  support  staff  to  assist  the  faculty. 

Representatives  from  six  schools  of  various  sizes  and  geographic  locations  felt  that 
assessments  reinforce  educational  integrity  and  should  be  the  foremost  consideration. 
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Additionally,  the  process  of  creating  an  academically  sound  assessment  is  difficult. 
Representatives  from  two  universities  expressed  doubt  as  to  whether  the  faculty  were  as  capable 
of  developing  adequate  assessments  as  they  should  be. 

Representatives  from  eight  of  the  universities  indicated  that  delivering  distance  education 
courses  is  labor-intensive  for  the  faculty,  in  particular,  for  courses  with  projects  requiring 
frequent  interaction  between  the  instructors  and  the  students.  Consequently,  several  schools 
experienced  difficulty  enticing  faculty  to  teach  these  courses.  The  faculty  resisted  the  additional 
workload  in  teaching  and  giving  exams.  But  several  schools  also  indicated  that  if  the  faculty 
could  be  convinced  of  the  time  savings  that  comes  with  using  technology  in  the  assessment 
process,  the  faculty  might  be  more  willing  to  use  electronic  assessments  in  the  distance  education 
courses. 

Representatives  from  four  schools  explicitly  denounced  the  electronic,  multiple  choice 
exams  as  not  being  representative  of  the  pedagogical  changes  that  are  possible.  The  faculties 
from  these  schools  were  sensitive  to  the  issue  of  simply  replacing  a  paper-based  exam  with  an 
electronic  exam.  The  faculties  advocated  a  new  look  at  the  curriculum  content,  an  evaluation  of 
the  technological  changes  possible  in  teaching,  and  the  technological  changes  that  are  possible 
for  assessments.  Representatives  from  two  highly  ranked  schools  said  that  the  technology 
changes  are  allowing  them  to  do  things  they  never  could  have  done  before. 

Proposition  3 

P3:  New  technology  provides  an  unknown  and  unproven  capability  to  the  institution, 
but  as  the  technology  becomes  more  prevalent,  an  institution  may  alter  the  types  of 
assessments  such  that  newer  technology  will  be  utilized. 

The  proposition  is  supported.  Representatives  from  nine  of  the  schools  expressed  strong 
apprehension  toward  using  technology-enabled  assessments.  Prevalence  of  computing 
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technology  for  the  students  and/or  faculty  is  an  important  concern  of  fourteen  schools. 
Interestingly,  the  security  aspects  of  the  new  technology  are  causing  the  initial  apprehensiveness 
as  opposed  to  the  cost  or  the  acquisition  process  in  the  university. 

The  two  most  significant  issues  in  this  area  are  concerns  that  the  students  must  have 
access  to  sufficient  technological  capabilities  and  the  faculty  must  receive  training  to  use  the 
technologies.  Representatives  from  seven  schools  indicated  technology  was  driving,  or  at  least 
encouraging,  universities  toward  electronic  assessments.  These  schools  varied  in  geographic 
location  and  size.  Representatives  from  several  of  the  remaining  universities  shared  the  idea  but 
were  less  enthusiastic.  One  recurring  limitation  mentioned  by  the  schools’  representatives  was  in 
the  area  of  laboratory  sciences.  The  concern  is,  essentially,  how  can  one  access  laboratory 
performance  at  a  distance?  One  remedy  was  to  take  a  laboratory  kit  to  a  site  and  complete  the 
assessment. 

Representatives  from  eleven  schools  specifically  addressed  the  technological  limitations 
of  the  students.  Financially,  a  student  is  less  able  to  acquire  sufficient  computing  capability  in 
terms  of  hardware,  software,  and  communications  infrastructure  than  a  university. 

Consequently,  there  are  limitations  to  the  current  assessment  capabilities.  This  limitation 
mandates  schools  to  conduct  electronic  assessments  via  the  lowest  technological  denominator  of 
the  students,  sites,  or  laboratories  involved. 

The  second  area  of  concern  is  in  faculty  training.  Representatives  from  seven  of  the 
schools  indicated  that  their  faculties  were  inadequately  experienced  in  the  use  of  technologies 
such  as  authoring  software,  electronic  mail,  or  interactive  chat  rooms.  Representatives  from 
three  of  the  schools  said  many  of  the  faculty  members  did  not  have  computers.  In  addition, 
many  of  those  having  computers  were  connected  to  neither  a  local  network  nor  the  Internet. 
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Representatives  from  the  same  schools  also  indicated  there  was  insufficient  communications 
infrastructure  on  a  campus-wide  scale.  Representatives  from  four  other  schools,  however, 
indicated  that  they  have  abundant  resources,  sufficient  communications  infrastructure,  and 
adequately  trained  faculty.  The  cause  of  the  disparity  is  out  of  the  scope  of  this  research  but,  the 
size  and  academic  rankings  were  very  different  between  the  two  groups. 

There  is  also  a  smaller  group  consisting  of  one  medium  southern  school  and  a  medium 
eastern  school  that  are  not  sure  the  technologies,  including  the  Internet,  are  sufficiently  mature  to 
depend  on  them  for  course  instruction  or  assessments.  One  faculty  indicated  several  experiences 
had  caused  mistrust  of  the  technology  by  the  school. 

Proposition  4 

P4:  Current  assessments  have  known  costs  and  are  likely  to  be  favored  unless  costs  of 
an  alternate  assessment  are  sufficiently  lower  to  alter  the  affordability  and  desirability  of 
the  current  assessment. 

The  data  from  the  interviews  weakly  supports  this  proposition.  A  representative  from 
only  one  school,  a  small  northern  school,  indicated  it  could  not  afford  the  costs  of  alternative 
assessments.  The  remaining  representatives,  each  to  varying  degrees,  indicated  the  cost  of 
assessments  is  generally  the  least  significant  issue  for  assessments.  In  fact,  the  representative 
from  one  school  said  cost  might  be  what  is  heard  about  most  but  is  the  least  important  issue. 
However,  four  schools  specifically  mentioned  the  long-term  cost  savings  as  compelling  the 
schools  to  pursue  different  assessments. 

Although  cost  is  relatively  insignificant,  two  main  issues  dominated  the  discussions. 

They  were  costs  related  to  faculty  compensation  and  the  fact  that  some  distance  education 
departments  receive  limited  public  funds.  The  cost  issues  relating  to  the  faculty  compensation 
scheme  was  discussed  most  often.  Representatives  from  three  schools  talked  about  the  cost  of 
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developing,  refining  an  assessment,  and  validating  assessments.  But,  representatives  from  four 
schools  felt  the  new  assessments  would,  in  the  long-run,  lower  overhead  costs,  decrease  postage 
expenses,  and  substantially  reduce  the  instructor-student  time  factor  in  distance  education  course 
assessments. 

A  representative  from  one  school  indicated  the  method  of  compensating  the  faculty 
prevented  the  creation  of  an  economical,  electronic  course.  The  school  pays  the  instructors  on  a 
per-term  basis.  That  is,  an  instructor  creates  and  conducts  an  electronic  course  for  a  specific 
term.  The  course  is  the  faculty’s  property  and  must  be  paid  again  to  create  and  deliver  the  course 
in  the  future.  Another  school  remedied  this  situation  by  paying  a  faculty  member  to  create  a 
course.  Then  each  time  the  course  is  used,  a  faculty  member  is  hired  to  deliver  the  course. 

Representatives  from  four  small  and  medium  universities  stated  that  distance  education 
funding  was  limited,  in  whole  or  in  part,  to  receipts  from  enrollments  in  the  distance  education 
courses.  This  funding  model  creates  a  situation  where  courses  are  developed  to  satisfy  a  specific 
demand.  Satisfying  the  demand  makes  it  incumbent  on  the  staff  to  conduct  the  market  research 
to  ensure  the  course  and  assessment  will,  in  fact,  generate  sufficient  enrollments  to  cover 
development  and  delivery  costs.  Obviously,  the  university  must  be  able  to  deliver  the  course  and 
assessments  within  a  reasonable  cost  range  or  it  may  price  itself  out  of  the  market. 

Proposition  5 

P5:  Inertia  in  an  institution  will  reinforce  the  current  assessments  used  within  the 
institution  and  will  not  alter  from  this  course  unless  technology  or  cost  issues  provide 
sufficient  motivation  to  do  so. 

This  proposition  is  strongly  supported  by  the  universities  in  this  research. 

Representatives  from  sixteen  of  the  universities  suggested  that  faculty  will  not,  in  varying 
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degrees,  support  changes  to  pedagogy  or  the  assessment  process.  Faculty  in  the  remaining  four 
schools  are  encouraged  and  motivated  to  utilize  new  forms  of  student  assessments. 

A  representative  from  one  medium  southern  school  indicated  that  the  faculty  did  not  see  a 
way  to  chart  a  new  course.  Consequently,  the  faculty  is  content  with  leaving  the  curriculum 
alone  and  letting  the  school  wait  until  they  retire  before  changing.  Unfortunately,  this  is  not  an 
isolated  opinion  in  the  universities  sampled.  Tenured  faculty  are  concerned  that  the  rules  that 
they  agreed  to  are  changing  without  their  control.  The  faculty’s  concern  adds  to  the  resistance  to 
change,  whether  the  change  is  positive  or  negative. 

A  representative  from  a  medium  sized  eastern  school  states  the  faculty  is  very  traditional 
and  they  want  to  keep  things  just  as  they  have  always  been.  Further,  a  representative  from  a 
large  mid-western  school  said  that  the  faculty  has  the  tendency  to  stick  with  the  tried  and  true 
methods  instead  of  venturing  into  the  unknown. 

A  representative  from  a  medium  sized  eastern  school  indicated  it  is  happy  with  traditional 
assessment  methods.  The  representative  added  that  it  has  found  little  reason  to  replace  paper 
assessments  with  electronic  assessments.  Representatives  from  two  universities  said  it  was  a 
tough  job  to  get  the  faculty  acclimated  to  new  technology.  A  representative  from  one  large 
northern  university  said  the  university  does  not  say  that  it  will  not  change,  it  just  does  not.  The 
same  representative  revealed  that  if  the  faculty  itself  wanted  the  change,  then  it  would  change. 

In  some  instances,  members  of  the  faculty  are  attempting  to  incorporate  technology  into 
the  courses.  However,  the  attempt  is  proving  to  be  problematic.  In  one  instance,  a  professor 
conducted  the  same  course  on-campus  and  via  distance  education.  The  instructor  had  several 
readings  on  reserve  at  the  library  for  the  on-campus  students.  When  the  instructor  taught  the 
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course  via  distance  learning,  the  materials  on  reserve  at  the  library  were  only  available  to  local 
students. 

Summary 

The  interview  data  rather  clearly  delineate  two  groups  of  schools.  One  group  tends  to  be 
more  reluctant  to  incorporate  new  technologies  into  the  assessment  process,  primarily  because  of 
security  concerns.  The  second  group  tends  to  incorporate  new  technologies  by  mitigating  the 
security  risks  associated  with  remote,  electronic  exams.  However,  representatives  from  schools 
in  both  groups  discussed  the  same  issues.  Essentially,  the  focus  and  relative  importance  were 
different. 

Overall,  the  propositions  were  supported  though  proposition  four  was  only  weakly 
supported.  In  general,  the  propositions  supported  the  ASM  indicating  that  although  the  literature 
review  did  not  specifically  address  the  issues,  it  did,  in  aggregate,  provide  insights  into  what 
universities  would  view  as  significant  issues. 

The  next  chapter  concludes  the  research  by  reviewing  the  research  questions,  providing 
additional  findings,  making  recommendations,  and  indicating  avenues  for  future  research. 
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V.  Conclusions 


Introduction 

This  study  looked  into  the  issues  that  influence  a  university’s  decision  making  process 
concerning  student  assessments  in  distance  education  curricula.  Primarily,  this  study  provides 
insights  into  the  fact  that  many  universities  use  advanced  technologies  to  deliver  course 
instruction  and  materials  but  do  not,  as  a  rule,  use  these  same  technologies  to  gather  assessments. 

The  background  literature  details  the  long  tradition  of  distance  education  and  its  place  in 
the  academic  world.  As  technology  has  matured  over  the  years,  so  has  the  delivery  methods  for 
course  instruction.  Today,  satellites  and  the  Internet  are  the  latest  delivery  methods  in  the 
technology  arena.  Within  these  two  delivery  conduits,  there  are  a  variety  of  implementation 
innovations  that  some  universities  are  exploring.  These  new  technologies  are  used  to  deliver 
course  instruction  and  some  universities  use  the  technologies  to  conduct  student  assessments. 
However,  many  universities  do  not.  Because  the  literature  does  not  address  why  many 
universities  do  not  conduct  student  assessments  using  these  new  technologies,  this  research  was 
designed  to  find  out  why. 

Review  of  Current  Literature 

From  the  current  literature,  the  study  identified  essential  constructs  for  creating  an 
assessment  selection  model.  The  literature  concerning  assessments  covered  a  wide  range  of 
issues,  but  did  not  specifically  reveal  the  factors  that  institutions  are  concerned  with  when 
deciding  the  acceptable  types  and  delivery  methods  for  assessments.  The  literature  offers  a  great 
deal  of  information  on  the  methods  to  develop,  validate,  and  use  assessments.  The  literature  also 
discusses  the  positive  and  negative  aspects  of  the  various  types  of  assessment  questions 


68 


including  ways  to  improve  assessments.  Additionally,  the  literature  discloses  research  on  the  use 
of  computers  in  the  classroom.  However,  the  research  discusses  computer  usage  as  it  benefits 
instruction  or  usage  as  a  training  tool  as  opposed  to  student  assessments  (Spencer,  1996;  Crumb, 
1990).  The  research  also  reflects  faculty  computer  use  as  essentially  automating  current 
instruction  methods  and  not  affecting  pedagogy. 

The  literature  provided  29  issues  in  various  subjects  and  degrees  of  applicability  to  the 
current  research.  The  29  issues  were  evaluated  and  categorized  into  five  broad  categories.  The 
categories  are  technology  issues,  administrative  issues,  inertia  issues,  security  issues,  and  cost 
issues. 

Technology  issues  are  concerned  with  the  technological  capabilities  available  to  the 
institution  to  deliver  the  assessment  to  the  student.  The  technology  issues  are:  future  use  of 
technology;  sufficient  technical  knowledge;  Internet  access;  physical  transmission  of  assessment; 
and  web-capable  software. 

Administrative  issues  deal  with  the  faculty’s  time  and  efforts  to  create,  maintain,  deliver, 
and  score  an  assessment.  The  administrative  issues  are:  faculty  perception  of  computer-based 
testing;  assessment  development,  maintenance,  and  scoring;  course  material  coverage; 
assessment  difficulty;  delivery  methods;  validity  and  reliability. 

Inertia  is  the  tendency  of  an  institution  to  resist  changes  and  maintain  the  status  quo. 
These  issues  also  encompass  the  institution's  concern  with  enhancing  or  maintaining  its 
academic  reputation.  The  inertia  issues  are:  pedagogy  changes;  reputation;  strategic  vision; 
academic  rigor;  standards;  institution  politics. 

Security  issues  are  concerned  with  the  institution  providing  a  secure  environment  for  the 
creation,  storage,  delivery,  and  scoring  of  an  assessment.  The  security  issues  are:  security 
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software  and  hardware  configurations;  password  and  restrict  assess;  assessment  security; 
physical  environment;  cheating. 

The  last  category,  cost,  addresses  those  issues  that  are  associated  with  the  assessment 
costs  and  the  costs  required  to  provide  the  assessment  to  the  students.  The  cost  issues  are: 
delivery  costs;  hardware  and  software  costs;  assessment  development  and  maintenance  costs; 
costs  to  implement  strategic  plan. 

Methodology 

This  is  an  exploratory  study  primarily  because  this  problem  is  not  described  in  the 
literature.  Consequently,  the  approach  was  to  develop  a  useful  model  (ASM)  for  describing  the 
assessment  selection  behavior  of  academic  institutions.  The  study  then  produced  the  descriptive 
data  required  to  evaluate  the  Assessment  Selection  Model ,  located  in  Figure  1  in  Chapter  II. 

Interview  questions,  located  at  Appendix  A,  were  developed  to  query  the  selected 
institutions  for  issues  about  the  assessment  selection  process.  The  telephone  interviewing 
technique  was  employed  to  elicit  the  data.  This  technique  produces  rich  data  very  appropriate 
for  this  type  of  study.  The  sample  was  selected  because  they  are  normal  subjects  not  indicative 
of  extreme  tendencies  or  views.  Of  the  55  universities,  20  were  interviewed.  To  ensure 
anonymity  of  the  participants,  the  identities  of  the  schools  interviewed  are  not  identified. 

Research  Questions 

The  research  questions  postulated  at  the  beginning  of  the  research  were  answered  during 
the  study.  The  data  supported  the  five  propositions  which  in  turn  supported  the  ASM.  The 
research  questions  and  answers  are  as  follows: 


70 


Research  Question  1 


Does  the  ASM  provide  insights  into  the  specific  issues  influencing  an  institution's 
decision  on  the  type  of  student  assessments  and  the  acceptable  delivery  method?  The 
Assessment  Selection  Model,  shown  in  Figure  1  in  Chapter  II,  proved  to  be  useful  in  capturing 
the  concerns  of  the  universities.  Further,  the  model  encompassed  the  realm  of  interest  from  the 
universities’  perspectives. 

Research  Question  2 

Are  there  additional  issues  that  institutions  are  considering  that  the  ASM  did  not  address? 
The  Assessment  Selection  Model  addressed  the  issues  that  the  universities  discussed  during  the 
interviews.  When  asked,  the  universities  but  did  not  provide  additional  issues  to  incorporate  into 
the  model. 

Research  Question  3 

What  is  the  relative  importance  among  the  issues  that  institutions  feel  are  significant? 

The  data  indicated  that  there  are  two  answers  to  this  question.  The  responding  universities  are 
grouped  into  two  divergent  groups  essentially  divided  on  the  intensity  of  the  security  issue.  One 
group  could  be  characterized  as  significantly  risk-averse.  Security  concerns  dictated,  in  essence, 
the  limitations  of  the  potential  types  of  assessment  types  and  delivery  that  would  be  used.  The 
other  group  is  characterized  as  risk-neutral  to  slightly  risk-averse.  This  group  tried  new  methods 
of  conducting  assessments,  mitigated  security  concerns  to  the  extent  consistent  with  on-campus 
courses,  and  tended  to  take  a  pedagogical  view  of  assessing  learning. 
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Additional  Findings 


Reputation 

Although  the  researcher  anticipated  concern  over  reputation  would  be  the  basis  of  the 
resistance,  it  actually  seems  to  be  tradition  and  school  size.  Based  on  the  demographics  of  the 
schools  interviewed,  the  larger  the  school  the  more  lethargic  the  faculty.  Specifically,  schools 
with  about  20,000  or  more  undergraduate  students  were  particularly  affected  by  this 
phenomenon. 

Interestingly,  the  schools  ranked  among  the  highest,  academically,  in  the  country  were 
the  least  concerned  with  the  distance  education  program’s  reputation  in  regards  to  unproctored 
exams  for  distance  education  courses.  The  interviews  revealed  that  from  their  perspective  the 
university  and  faculty  reputation  provided  the  solid  assurance  that  quality  and  academic 
standards  would  be  maintained.  Therefore,  their  solid  reputation  allowed  them  to  conduct 
distance  education  courses  and  unproctored  assessments  without  the  fear  of  diminishing  or 
tarnishing  that  reputation.  These  schools  were  more  likely  to  and  actually  did  conduct 
unproctored  exams. 

Other  universities  that  ranked  lower  in  the  country,  academically,  seem  to  focus  more  on 
the  potential  damage  to  the  reputation  of  the  university  if  unproctored  exams  were  used.  The 
interviews  indicated  that  these  schools  must  be  not  only  sure,  but  inordinately  sure  no  possibility 
exists  for  someone  to  cheat  on  an  exam.  Consequently,  many  opportunities  to  alter  processes  to 
accommodate  students  or  faculty  will  be  curtailed  or  completely  avoided  simply  because  of  this 
concern. 

In  essence,  there  seems  to  be  an  inverse  relationship  between  academic  reputation  and  the 
school’s  concern  to  conduct  distance  education  courses  and  assessments  in  a  certain  way  to 
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prevent  a  diminished  opinion  of  the  reputation  of  the  school.  For  this  study,  the  stronger  the 
reputation,  the  less  preoccupied  the  school  will  be  that  unproctored  assessments  will  detract  from 
the  school’s  reputation.  On  the  other  hand,  a  lessened  reputation  correlated  with  an  increased 
concern  that  unproctored  assessments  would  detract  from  the  school’s  reputation. 

Security  Standards 

Security  concerns  also  related  to  the  reputation  issue.  However,  the  security  standard 
discovered  in  the  research  is  the  tendency  for  distance  education  programs  to  implement 
dramatically  more  stringent  security  standards  than  those  used  for  on-campus  students. 

Typically,  on-campus  courses  required  students  to  present  a  picture  identification  card  to  take  an 
exam.  Identification  cards  can  and  have  been  created  so  that  students  can  cheat  for  one  another. 
In  classrooms  or  lecture  halls  with  many  students,  the  faculty’s  ability  to  distinguish  the  true 
students  from  a  proxy  is  excessively  difficult.  The  ability  to  distinguish  becomes  increasingly 
problematic  in  a  traditional  one-way  lecture  format  course.  With  little  or  no  student-professor 
interaction,  a  professor  will  not  necessarily  know  the  student  by  name  or  ability.  Some  distance 
education  programs  focuses  on  creating  rules  and  procedures  to  completely  eliminate  cheating. 
The  normal  method  is  to  use  one  or  more  proctors. 

The  interviews,  as  a  whole,  indicated  the  most  important  security  concern  is  to  know, 
with  certainty,  who  is  at  the  other  end  of  the  keyboard.  This  is  akin  to  knowing,  with  certainty, 
who  is  sitting  in  each  and  every  seat  in  the  lecture  hall.  Knowing  this  is  desired  but  not  entirely 
possible.  The  university  is  accustomed  to  accepting  a  reduced  level  of  risk  by  requiring  picture 
identification  cards.  However,  some  universities  are  not  so  inclined  to  use  that  reasoning  in  the 
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context  of  electronic  assessments.  For  example,  instead  of  a  student  presenting  a  picture 
identification  card,  the  student  uses  a  password  and  login. 

Not  all  universities  share  the  same  level  of  anxiety  concerning  this  issue.  Some  were 
content  with  knowing  that  adequate  precautions  were  in  place.  Others  will  not  be  content  until 
the  student’s  environment  is  entirely  secured.  The  level  of  contentment  illustrates  a  dichotomous 
relationship  as  opposed  to  a  fluid  scale  between  the  sample  of  universities.  Eleven  of  the  schools 
gravitated  toward  mitigating  the  risks  and  progressing  as  compared  to  nine  of  the  schools 
refusing  to  accept  the  risks  and  consequently  maintained  the  current  methodology. 

Mitigating  the  Risks 

All  of  the  universities  mitigated  known  security  risks  by  providing  methods  to  assure  the 
student’s  efforts  and  identity.  The  schools  most  concerned  with  the  student’s  identity  and  origin 
of  work  used  proctors  to  mitigate  the  risks  of  the  unknown. 

The  universities  that  were  less  preoccupied  with  the  student’s  identity  and  verification  of 
the  work  are  using  various  methods  to  reduce  the  faculty’s  concern.  This  lack  of  preoccupation 
should  not  be  construed  to  mean  that  the  faculties  at  these  universities  are  not  significantly 
concerned.  They  simply  acknowledge  the  risks  and  use  acceptable  techniques  to  mitigate  the 
risks.  As  previously  mentioned,  this  mitigation  strategy  is  the  same  method  that  faculty  use  for 
on-campus  courses.  They  accept  that  a  chance  exists  for  a  student  to  take  an  exam  for  another 
student.  Consequently,  a  picture  identification  card  is  used  to  reduce  the  incidences  of  cheating 
and  to  raise  the  level  of  efforts  required  by  the  student  to  cheat. 

One  school  has  a  requirement  for  at  least  three  telephone  calls  between  the  faculty  and 
each  student  during  the  distance  education  course.  The  telephone  contact  creates  a  more 
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personal  relationship  between  the  instructor  and  student,  thereby  reducing  the  tendency  for  the 
student  to  cheat.  The  telephone  contact  also  adds  to  the  instructor’s  knowledge  about  the  student 
and  will  give  the  instructor  a  better  feel  as  to  the  student’s  abilities. 

Another  school  utilizes  frequent  electronic  communications  between  the  instructor  and 
students  during  the  course.  There  are  frequent  electronic  classroom  meetings  where  students 
share  information  on  the  topics  of  the  course.  The  instructor  has  an  electronic  record  of  the 
communications  from  each  of  the  students.  As  the  course  progresses,  the  instructor  is  better  able 
to  discern  the  student’s  ability.  Accordingly,  the  instructor  recognizes  projects  or  documents  that 
are  out  of  the  student’s  range  of  abilities.  The  underlying  assumption  is  the  student  will  not 
dramatically  alter  grammatical  usage,  intellectual  approach  to  completing  a  project,  or  problem 
solving  skills.  In  this  way,  the  instructor  can  estimate  whether  students  are  submitting  their  own 
work.  This  technique  assumes  that  the  class  size  is  sufficiently  small  to  allow  that  level  of  effort 
by  the  instructor. 

A  similar  situation  exists  at  another  school.  This  school  believes  that  frequent 
communication  between  students  and  instructors  encourages  the  students  to  provide  their  own 
work.  The  representative  from  the  university  conceded  that  someone  may  cheat  on  an  exam  and 
go  undetected,  but  it  is  unlikely  that  the  student  could  get  someone  to  be  a  substitute  during  an 
entire  course.  The  representative  also  indicated  that  it  is  unlikely  that  a  student  can  get  others  to 
cheat  for  the  student  for  the  duration  of  the  degree  program. 

Some  universities  use  group  and  individual  projects,  papers,  and  exams  as  evaluation 
mechanisms  for  courses.  The  universities  that  used  electronic  exams  tended  to  not  use  multiple 
choice  exams.  The  schools  normally  use  questions  that  require  thought  and  interpretation  in 
contrast  with  questions  requiring  recognition  or  memorization  of  the  course  material.  The 
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interpretation  questions  will  require  the  student  to  inject  class  experiences  or  life  experiences  to 
personalize  the  answers.  By  requiring  more  substantive  answers,  it  lessens  the  possibility  for 
students  to  use  substitutes  or  receive  assistance. 

Faculty 

With  few  exceptions,  the  faculty  were  described  as  being  resistant  to  technology  and 
generally  unwilling  to  move  forward  in  that  regard.  The  tendency  to  resist  technology  was  less 
pronounced  in  private  as  opposed  to  the  public  schools.  Additionally,  as  indicated  by  the 
representatives  from  the  six  southern  schools,  the  resistance  was  much  more  pronounced  among 
southern  universities. 

Representatives  from  several  schools  identified  methods  that  were  being  used  to  reverse 
this  trend.  Administration  from  one  school  required  each  new  faculty  member  to  develop  an 
electronic  course  and  learn  the  technology  involved.  In  this  way,  the  administration  hoped  to 
reduce  the  resistance  to  new  methods  of  approaching  the  educational  experience. 

Intellectual  property  rights  was  also  an  issue  that  seems  to  be  critically  important  to  the 
faculty  at  many  of  the  public  universities,  especially  tenured  faculty.  Representatives  from 
several  universities  mentioned  that  attracting  new  faculty  to  distance  education  courses  was 
probably  difficult  because  of  this  issue.  Faculty  compensation  models  are  closely  related  to  this 
issue.  Representatives  from  one  institution  disclosed  that  they  have  significant  problems  with 
tenured  faculty  refusing  to  create  electronic  courses  because  of  the  loss  of  compensation  for  the 
subsequent  uses  of  the  course.  The  same  representative  indicated  that  faculty  must  be  frequently 
reassured  that  the  course  is  their  domain. 
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Recommendations 


The  research  suggests  that  there  is  no  single,  best  way  to  conduct  assessments. 
Composition  of  the  student  population  is  a  significant  factor.  Representatives  from  three  schools 
mentioned  that  students  of  different  age  groups  demonstrate  dissimilar  levels  of  integrity  and 
professionalism.  Distance  education  courses  have  an  average  student  age  above  that  of  on- 
campus  courses,  although  the  gap  is  narrowing.  In  addition,  the  technology  available  to  students 
within  a  university  and  the  technology  available  to  students  of  different  universities  can  limit 
each  school’s  infusion  of  technology  into  its  distance  education  program. 

However,  the  methods  that  other  universities  used  to  mitigate  the  risks  should  be 
evaluated  for  possible  incorporation  into  the  program  of  each  school.  This  idea  is  similar  to  a 
best-practices  scenario.  Nevertheless,  the  culture  and  faculty  tendencies  of  the  school  are  a 
major  consideration  for  making  any  changes  or  suggestions.  Assessments  in  the  distance 
education  program  should  balance  convenience,  academic  standards,  and  ability  of  the 
assessment  to  elicit  the  student’s  true  mastery  of  the  material. 

Limitations  of  the  Research 

The  level  of  literature  covering  this  particular  subject  limits  this  exploratory  study. 
Consequently,  the  Assessment  Selection  Model  may  be  insufficient  to  address  the  issues  of 
concern  for  universities’  assessment  selection  decisions. 

This  research  selected  a  subset  of  universities  listed  in  two  books.  A  fundamental 
assumption  of  the  study  is  that  the  fifty-five  selected  universities  are  representative  of  the  greater 
population  of  universities  offering  distance  education.  Additionally,  because  data  collection  was 
limited  to  twenty  interviews,  generalizability  is  limited. 
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Finally,  because  the  study  was  conducted  by  a  single  researcher,  biases  could  be  present. 
Furthermore,  the  personal  interviewing  process  may  have  inadvertently  limited  or  focused  the 
answers  from  the  university  representatives  causing  the  study  to  receive  incomplete  information. 

Suggestions  for  Future  Research 

During  the  course  of  the  research,  there  were  areas  uncovered  that  require  additional 
research. 

First,  research  should  be  conducted  to  ascertain  the  actual  propensity  of  students  to  cheat 
in  distance  education  environments,  particularly  in  electronic  courses.  From  the  interviews, 
there  is  an  underlying  assumption  that  students  will  cheat.  This  assumption  seems  to  be  a 
significant  presumption  that  may  or  may  not  be  substantiated.  The  typical  demographic  data  on 
a  distance  education  student  is  markedly  different  than  an  on-campus  student.  There  are  studies 
on  student  cheating,  but  they  focus  on  students  in  classrooms. 

Secondly,  many  universities  were  concerned  with  the  computing  capabilities  and  abilities 
of  the  students.  Capabilities  refer  to  the  hardware  and  software  equipment  and  the 
telecommunications  connectivity  available  for  the  student,  whereas  the  abilities  of  the  student 
refer  to  the  student’s  aptitude  and  ability  to  use  the  technology  for  the  course  and  assessment. 

The  school  is  forced  to  the  lowest  common  denominator  of  the  group  of  students  the  university 
has  as  its  target  population.  Therefore,  research  into  the  capabilities  and  abilities  of  distance 
education  students  would  benefit  the  universities  at  large. 

Finally,  this  research  looked  into  accredited,  academic  universities  and  their  perspectives. 
Research  should  be  directed  at  schools  with  training  programs  to  reveal  their  perceptions  about 
the  same  issues  or  determine  if  they  have  different  issues  affecting  them.  From  a  cursory  look 
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into  these  schools,  they  seem  more  liberal  in  the  area  of  security  requirements.  It  would  be 
interesting  to  identify  what  mitigating  techniques  they  used  for  the  risks  involved. 
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Appendix  A:  Interview  Form 


Introduction 
of  Author: 

My  name  is  Clayton  Sammons.  I  am  a  graduate  student  at  the  Air  Force 
Institute  of  Technology  in  Dayton  Ohio. 

Purpose: 

I  am  conducting  research  concerning  institutions’  decision-making 
policies  with  regards  to  choosing  individual  student  assessments  used  in 
the  distance  education  course  curriculum.  The  information  that  you  give 
me  will  be  used  to  test  a  model  of  an  institution’s  assessment  selection 
behavior.  I  ensure  your  anonymity  for  all  of  your  responses  and  will  not 
specifically  mention  your  name  in  the  research  document.  I  will  also 
provide  you  an  executive  summary  of  the  findings,  if  you  wish. 

Overview  of 
Interview: 

There  are  two  sections  of  questions  for  the  interview.  Section  one 
contains  general  questions  about  your  institution  and  section  two  has 
specific  questions  relating  to  assessments  and  issues  concerned  with 
various  aspects  of  assessments.  I  will  take  notes  of  your  responses 
during  the  interview  and  I  ask  for  your  patience  as  I  ensure  completeness. 

How  You 
Were 

Selected: 

You  were  selected  from  several  books  listing  academic  institutions 
conducting  various  types  of  distance  education.  From  that,  I  randomly 
selected  20  institution's  to  contact. 

Time 

Required: 

This  interview  should  take  no  longer  than  minutes. 

Time  of 
Interview: 

Began:  Stop: 

Day  of 
Interview: 

Attitude  of 
Interviewee: 

Cooperative 

Neither  cooperative  or  uncooperative 

Uncooperative 

Other 

Call  Back? 

If  this  is  not  a  good  time  to  talk,  I  can  call  back  at  a  more  convenient 
time. 

Date: 

Time: 
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If  Not  The  Name: 

Right  Person, 

Who?  Number: 

Position: 


Are  there 
Questions 
Before  We 
Begin? 


Receive  from  Full  name 
interviewee: 

Position 


Address 


E-mail  address 

Phone  number 

Institution 


Alternative  contact 

Years  in  academia 
Years  in  current  position 
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Section  1. 


First,  the  nine  general  questions  related  to  attributes  of  your  institution's  distance 
education  program. 

1 .  Does  your  institution  provide  distance  education  courses?  (Y/N) 

2.  What  kind  of  subject  areas  does  your  institution  offer  using  distance  education? 

3.  Does  your  institution  provide  distance  education  courses  via  the  Internet?  (Y/N)  Why  or  why 
not? 

4.  What  method  does  your  institution  use  to  distribute  course  information  for  the  distance 
education  courses?  (E-mail,  videotapes,  Internet,  teleconferencing,) 

5.  Does  your  institution  conduct  proctored  or  unproctored  assessments?  Proctored  assessments 
are  those  that  students  complete  under  the  direct  supervision  of  a  designated  or  approved 
individual.  Unproctored  assessments  are  those  that  the  student  completes  without  supervision. 
This  does  not  eliminate  restrictions  on  the  student  behavior  but  simply  means  that  the  student  is 
not  supervised  during  the  completion  of  the  exam. 

6.  Do  the  methods  of  delivering  course  assessments  correspond  to  the  delivery  method  of  the 
course  instruction?  For  example,  if  course  material  is  delivered  via  the  Internet,  is  the  student's 
course  assessment  also  delivered  via  the  Internet? 

7.  What  type  of  assessment  questions  does  your  institution  use  for  the  distance  education 
courses?  Possible  types  include  multiple  choice,  true/false,  essay,  short  answer,  group  projects, 
individual  projects,  etc. 

8.  Who  are  the  decision-makers  at  your  institution  that  establish  the  acceptable  assessment  types 
and  delivery  methods?  (Instructors,  deans,  board,  etc.) 

9.  Are  there  differences  between  the  acceptable  types  of  course  assessments  for  the  on-campus 
courses  and  the  distance  education  courses?  What  are  they?  Why  are  they  the  same  or  different? 
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Section  2. 


Secondly,  I  have  a  few  questions  related  to  the  decision-making  behavior  at  your 
institution  concerning  the  types  of  individual  student  assessments  and  delivery  methods  used  in 
your  institution's  distance  education  program.  I  need  you  to  think  about  your  institution  with  its 
programs  and  institutional  policies  and  procedures  used  to  make  decisions  about  assessments. 
For  this  research,  you  may  think  about  tests,  examinations,  and  assessments  interchangeably  as 
long  as  it  relates  to  a  single  student  in  a  course. 

1 .  What  are  the  issues  that  prevent  your  institution  from  attempting  innovative  assessment 
methods,  i.e.  maintain  the  status  quo? 

la.  Based  on  these  issues,  what  factors  prevent  your  institution  from  implementing  new  or 
different  assessments  types  or  delivery  methods? 

2.  What  are  the  technological  issues  or  limitations  that  your  institution  is  concerned  with  when 
deciding  what  types  of  assessments  will  be  approved  for  your  institution?  By  technological 
issues,  I  mean  those  issues  that  are  concerned  with  the  technological  capabilities  available  to  the 
institution  to  deliver  the  assessment  to  the  student. 

2a.  Based  on  your  technology  issues,  what  factors  prevent  your  institution  from  implementing 
new  or  different  assessment  types  or  delivery  methods? 

2b.  What  role  do  technology  issues  tend  to  have  in  your  institution  that  either  keep  the  same 
types  of  assessments  or  change  to  new  types  of  assessments? 

3.  What  are  the  cost  issues  or  limitations  that  your  institution  is  concerned  with  when  deciding 
what  types  of  assessments  will  be  approved  for  your  institution?  By  cost  issues,  I  mean  those 
issues  that  are  associated  with  the  costs  of  the  assessment  and  all  of  the  costs  required  to  provide 
the  assessment  to  the  students. 

3a.  Based  on  your  cost  issues,  what  factors  prevent  your  institution  from  implementing  new  or 
different  assessment  types  or  delivery  methods? 

3b.  What  role  do  cost  issues  tend  to  have  in  your  institution  that  either  keep  the  same  types  of 
assessments  or  change  to  new  types  of  assessments? 

4.  What  are  the  administrative  issues  or  limitations  that  your  institution  is  concerned  with  when 
deciding  what  types  of  assessments  will  be  approved  for  your  institution?  By  administrative 
issues,  I  mean  those  issues  that  encompass  the  faculty’s  time  and  efforts  to  create,  maintain, 
deliver,  and  score  an  assessment.  This  would  also  include  all  of  the  features  required  of  an 
academically  rigorous  assessment. 

4a.  Based  on  your  administrative  issues,  what  factors  prevent  your  institution  from 
implementing  new  or  different  assessment  types  or  delivery  methods? 
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4b.  What  role  do  administrative  issues  tend  to  have  in  your  institution  that  either  keep  the  same 
types  of  assessments  or  change  to  new  types  of  assessments? 

5.  What  are  the  security  issues  or  limitations  that  your  institution  is  concerned  with  when 
deciding  what  types  of  assessments  will  be  approved  for  your  institution?  By  security  issues,  I 
mean  those  issues  that  are  concerned  with  the  institution  providing  a  secure  environment  for  the 
creation,  storage,  delivery,  and  scoring  of  an  assessment. 

5a.  Based  on  your  security  issues,  what  factors  prevent  your  institution  from  implementing  new 
or  different  assessment  types  or  delivery  methods? 

5b.  What  role  do  security  issues  tend  to  have  in  your  institution  that  either  keep  the  same  types 
of  assessments  or  change  to  new  types  of  assessments? 

6.  Are  there  any  other  issues  that  affect  the  selection  decisions  at  your  institution  that  have  not 
already  been  covered? 

7.  Of  all  the  issues  that  you  have  discussed,  which  are  the  most  important  to  your  institution? 
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Appendix  B:  Sample  Population 


1 .  University  of  Alabama 

2.  University  of  Alaska 

3.  University  of  Arizona 

4.  University  of  Arkansas  at  Little  Rock 

5.  University  of  California 

6.  Colorado  State  University 

7.  University  of  Delaware 

8.  Empire  State  College 

9.  University  of  Florida 

10.  University  of  Georgia 

1 1 .  Georgia  Institute  of  T echnology 

12.  University  of  Idaho 

13.  Indiana  State  University 

14.  Indiana  University 

1 5 .  University  of  Iowa 

16.  University  of  Kansas 

1 7.  Kansas  State  University 

1 8.  Louisiana  State  University 

19.  University  of  Maine  at  Augusta 

20.  University  of  Maryland 

2 1 .  Western  Michigan  University 

22.  University  of  Minnesota 

23.  University  of  Missouri 

24.  University  of  Montana 

25.  University  of  Nebraska  at  Lincoln 

26.  University  of  Nevada 

27.  University  of  New  Hampshire 

28.  New  Jersey  Institute  of  Technology 

29.  New  School  for  Social  Research 

30.  New  York  University 

3 1 .  University  of  North  Carolina 

32.  University  of  North  Dakota 

33.  Nova  University 

34.  Ohio  State  University 

35.  Ohio  University 

36.  University  of  Oklahoma 

37.  Oklahoma  State  University 

38.  Old  Dominion  University 

39.  Pennsylvania  State  University 

40.  University  of  Phoenix  -  San  Francisco 

41.  Portland  State  University 

42.  Purdue  University 

43 .  Rochester  Institute  of  T echnology 
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44.  University  of  South  Carolina 

45 .  University  of  South  Dakota 

46.  University  of  Southern  Colorado 

47.  University  of  Tennessee 

48.  Texas  Tech  University 

49.  University  of  Texas 

50.  University  of  Utah 

5 1 .  University  of  Washington 

52.  Washington  State  University 

53.  University  of  Wisconsin 

54.  University  of  Wyoming 

55.  Brigham  Young  University 
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